Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250628 07:52:08.795722 3438 test_util.cc:276] Using random seed: 307510712
W20250628 07:52:10.022774 3438 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.184s user 0.432s sys 0.750s
W20250628 07:52:10.023252 3438 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.184s user 0.432s sys 0.750s
I20250628 07:52:10.025676 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:52:10.025942 3438 ts_itest-base.cc:116] --------------
I20250628 07:52:10.026145 3438 ts_itest-base.cc:117] 4 tablet servers
I20250628 07:52:10.026357 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:52:10.026568 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:52:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:52:10Z Disabled control of system clock
I20250628 07:52:10.072974 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43105
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:43105 with env {}
W20250628 07:52:10.387782 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:10.388448 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:10.388931 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:10.421675 3452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:52:10.422029 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:10.422307 3452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:52:10.422545 3452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:52:10.459053 3452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:43105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:10.460405 3452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:10.462244 3452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:10.477355 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:10.477463 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:11.723667 3452 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.250s user 0.495s sys 0.753s
W20250628 07:52:11.724121 3452 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.250s user 0.495s sys 0.753s
I20250628 07:52:11.724476 3452 server_base.cc:1048] running on GCE node
W20250628 07:52:11.725531 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:11.726984 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:11.729943 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:11.731360 3452 hybrid_clock.cc:648] HybridClock initialized: now 1751097131731315 us; error 74 us; skew 500 ppm
I20250628 07:52:11.732244 3452 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:11.745054 3452 webserver.cc:469] Webserver started at http://127.3.91.190:35873/ using document root <none> and password file <none>
I20250628 07:52:11.746135 3452 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:11.746471 3452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:11.746965 3452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:11.751966 3452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "118fcb588aae44da8422e64b17075376"
format_stamp: "Formatted at 2025-06-28 07:52:11 on dist-test-slave-wggc"
I20250628 07:52:11.753439 3452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "118fcb588aae44da8422e64b17075376"
format_stamp: "Formatted at 2025-06-28 07:52:11 on dist-test-slave-wggc"
I20250628 07:52:11.761855 3452 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250628 07:52:11.768846 3468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:11.770148 3452 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250628 07:52:11.770529 3452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "118fcb588aae44da8422e64b17075376"
format_stamp: "Formatted at 2025-06-28 07:52:11 on dist-test-slave-wggc"
I20250628 07:52:11.770885 3452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:11.856562 3452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:11.858155 3452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:11.858637 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:11.932796 3452 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:43105
I20250628 07:52:11.932901 3519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:43105 every 8 connection(s)
I20250628 07:52:11.935632 3452 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:52:11.941032 3520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:11.943310 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 3452
I20250628 07:52:11.943886 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:52:11.967366 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Bootstrap starting.
I20250628 07:52:11.975517 3520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:11.977972 3520 log.cc:826] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:11.983101 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: No bootstrap required, opened a new log
I20250628 07:52:12.001351 3520 raft_consensus.cc:357] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:12.002081 3520 raft_consensus.cc:383] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:12.002357 3520 raft_consensus.cc:738] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118fcb588aae44da8422e64b17075376, State: Initialized, Role: FOLLOWER
I20250628 07:52:12.003096 3520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:12.003626 3520 raft_consensus.cc:397] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:52:12.003893 3520 raft_consensus.cc:491] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:52:12.004190 3520 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:12.009308 3520 raft_consensus.cc:513] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:12.010216 3520 leader_election.cc:304] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118fcb588aae44da8422e64b17075376; no voters:
I20250628 07:52:12.012197 3520 leader_election.cc:290] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:52:12.013168 3525 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:12.015738 3525 raft_consensus.cc:695] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 LEADER]: Becoming Leader. State: Replica: 118fcb588aae44da8422e64b17075376, State: Running, Role: LEADER
I20250628 07:52:12.016577 3525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:12.017554 3520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:52:12.025794 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "118fcb588aae44da8422e64b17075376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } } }
I20250628 07:52:12.026054 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118fcb588aae44da8422e64b17075376. Latest consensus state: current_term: 1 leader_uuid: "118fcb588aae44da8422e64b17075376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } } }
I20250628 07:52:12.026638 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:12.026638 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:12.031365 3533 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:52:12.045557 3533 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:52:12.062737 3533 catalog_manager.cc:1349] Generated new cluster ID: 03bca85c0ee94834b0cca3a0fb9d34d6
I20250628 07:52:12.063009 3533 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:52:12.080224 3533 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:52:12.082056 3533 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:52:12.096819 3533 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Generated new TSK 0
I20250628 07:52:12.098248 3533 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:52:12.121398 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 07:52:12.459641 3544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:12.460202 3544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:12.460712 3544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:12.494012 3544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:12.494925 3544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:52:12.533097 3544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:12.534650 3544 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:12.536828 3544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:12.558549 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:12.562043 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:12.561048 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:12.561328 3544 server_base.cc:1048] running on GCE node
I20250628 07:52:13.800606 3544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:13.803794 3544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:13.805320 3544 hybrid_clock.cc:648] HybridClock initialized: now 1751097133805255 us; error 80 us; skew 500 ppm
I20250628 07:52:13.806248 3544 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:13.815171 3544 webserver.cc:469] Webserver started at http://127.3.91.129:38083/ using document root <none> and password file <none>
I20250628 07:52:13.816313 3544 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:13.816560 3544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:13.817090 3544 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:13.822459 3544 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "362055de6a1245e0b5e27db34739dc33"
format_stamp: "Formatted at 2025-06-28 07:52:13 on dist-test-slave-wggc"
I20250628 07:52:13.823876 3544 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "362055de6a1245e0b5e27db34739dc33"
format_stamp: "Formatted at 2025-06-28 07:52:13 on dist-test-slave-wggc"
I20250628 07:52:13.832684 3544 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20250628 07:52:13.839690 3560 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:13.841572 3544 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
I20250628 07:52:13.842073 3544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "362055de6a1245e0b5e27db34739dc33"
format_stamp: "Formatted at 2025-06-28 07:52:13 on dist-test-slave-wggc"
I20250628 07:52:13.842559 3544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:13.921557 3544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:13.923369 3544 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:13.923851 3544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:13.926843 3544 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:13.932085 3544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:13.932422 3544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250628 07:52:13.932716 3544 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:13.932886 3544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:14.087756 3544 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:34269
I20250628 07:52:14.087877 3672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:34269 every 8 connection(s)
I20250628 07:52:14.090425 3544 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:52:14.099825 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 3544
I20250628 07:52:14.100345 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:52:14.107621 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:52:14.118005 3673 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:14.118507 3673 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:14.119781 3673 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:14.123019 3485 ts_manager.cc:194] Registered new tserver with Master: 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269)
I20250628 07:52:14.126271 3485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:57259
W20250628 07:52:14.423179 3677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:14.423723 3677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:14.424187 3677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:14.456593 3677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:14.457497 3677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:52:14.494102 3677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:14.495404 3677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:14.497120 3677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:14.513303 3683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:15.130963 3673 heartbeater.cc:499] Master 127.3.91.190:43105 was elected leader, sending a full tablet report...
W20250628 07:52:14.514766 3686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:14.513317 3684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:15.726877 3685 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:52:15.726934 3677 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:52:15.730715 3677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:15.732908 3677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:15.734274 3677 hybrid_clock.cc:648] HybridClock initialized: now 1751097135734232 us; error 59 us; skew 500 ppm
I20250628 07:52:15.735168 3677 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:15.741480 3677 webserver.cc:469] Webserver started at http://127.3.91.130:44557/ using document root <none> and password file <none>
I20250628 07:52:15.742467 3677 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:15.742672 3677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:15.743108 3677 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:15.747774 3677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "319632b9d09f4ce9b2d72dda854b460f"
format_stamp: "Formatted at 2025-06-28 07:52:15 on dist-test-slave-wggc"
I20250628 07:52:15.748891 3677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "319632b9d09f4ce9b2d72dda854b460f"
format_stamp: "Formatted at 2025-06-28 07:52:15 on dist-test-slave-wggc"
I20250628 07:52:15.756232 3677 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250628 07:52:15.762369 3693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:15.763477 3677 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 07:52:15.763849 3677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "319632b9d09f4ce9b2d72dda854b460f"
format_stamp: "Formatted at 2025-06-28 07:52:15 on dist-test-slave-wggc"
I20250628 07:52:15.764168 3677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:15.831522 3677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:15.833127 3677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:15.833567 3677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:15.836244 3677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:15.840571 3677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:15.840812 3677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:15.841017 3677 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:15.841152 3677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:15.985260 3677 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:39249
I20250628 07:52:15.985375 3805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:39249 every 8 connection(s)
I20250628 07:52:15.987937 3677 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:52:15.991338 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 3677
I20250628 07:52:15.991954 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:52:15.998988 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:52:16.010246 3806 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:16.010851 3806 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:16.012290 3806 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:16.015304 3485 ts_manager.cc:194] Registered new tserver with Master: 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249)
I20250628 07:52:16.017354 3485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:46503
W20250628 07:52:16.316771 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:16.317268 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:16.317760 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:16.351400 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:16.352257 3810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:52:16.390532 3810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:16.392059 3810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:16.394023 3810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:16.412520 3817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:17.022058 3806 heartbeater.cc:499] Master 127.3.91.190:43105 was elected leader, sending a full tablet report...
W20250628 07:52:16.412808 3819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:16.412838 3816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:17.611656 3818 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1201 milliseconds
I20250628 07:52:17.612319 3810 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:52:17.616036 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:17.618337 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:17.619817 3810 hybrid_clock.cc:648] HybridClock initialized: now 1751097137619738 us; error 75 us; skew 500 ppm
I20250628 07:52:17.620702 3810 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:17.635192 3810 webserver.cc:469] Webserver started at http://127.3.91.131:44987/ using document root <none> and password file <none>
I20250628 07:52:17.636227 3810 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:17.636451 3810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:17.636924 3810 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:17.641696 3810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ddc625119ed24d268bd65a8422ffb6e4"
format_stamp: "Formatted at 2025-06-28 07:52:17 on dist-test-slave-wggc"
I20250628 07:52:17.642943 3810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ddc625119ed24d268bd65a8422ffb6e4"
format_stamp: "Formatted at 2025-06-28 07:52:17 on dist-test-slave-wggc"
I20250628 07:52:17.650907 3810 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 07:52:17.657325 3826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:17.658690 3810 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:52:17.659109 3810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ddc625119ed24d268bd65a8422ffb6e4"
format_stamp: "Formatted at 2025-06-28 07:52:17 on dist-test-slave-wggc"
I20250628 07:52:17.659478 3810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:17.714371 3810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:17.715927 3810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:17.716399 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:17.719195 3810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:17.723903 3810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:17.724130 3810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:17.724347 3810 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:17.724489 3810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:17.879748 3810 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:37951
I20250628 07:52:17.879891 3938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:37951 every 8 connection(s)
I20250628 07:52:17.882520 3810 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:52:17.892632 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 3810
I20250628 07:52:17.893064 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:52:17.899312 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.132:0
--local_ip_for_outbound_sockets=127.3.91.132
--webserver_interface=127.3.91.132
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:52:17.905731 3939 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:17.906212 3939 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:17.907552 3939 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:17.910440 3485 ts_manager.cc:194] Registered new tserver with Master: ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131:37951)
I20250628 07:52:17.911710 3485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:53171
W20250628 07:52:18.224644 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:18.225195 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:18.225733 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:18.257854 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:18.258702 3943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.132
I20250628 07:52:18.294190 3943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.91.132
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43105
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.132
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:18.295504 3943 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:18.297508 3943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:18.314987 3950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:18.916047 3939 heartbeater.cc:499] Master 127.3.91.190:43105 was elected leader, sending a full tablet report...
W20250628 07:52:18.315114 3952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:18.315201 3949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:18.317695 3943 server_base.cc:1048] running on GCE node
I20250628 07:52:19.494289 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:19.497057 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:19.498548 3943 hybrid_clock.cc:648] HybridClock initialized: now 1751097139498508 us; error 54 us; skew 500 ppm
I20250628 07:52:19.499327 3943 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:19.506137 3943 webserver.cc:469] Webserver started at http://127.3.91.132:46655/ using document root <none> and password file <none>
I20250628 07:52:19.507043 3943 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:19.507247 3943 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:19.507661 3943 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:19.512255 3943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "92878c5906f643a5aa9dafd7a1970f3f"
format_stamp: "Formatted at 2025-06-28 07:52:19 on dist-test-slave-wggc"
I20250628 07:52:19.513347 3943 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "92878c5906f643a5aa9dafd7a1970f3f"
format_stamp: "Formatted at 2025-06-28 07:52:19 on dist-test-slave-wggc"
I20250628 07:52:19.520690 3943 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20250628 07:52:19.526690 3960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:19.527766 3943 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250628 07:52:19.528069 3943 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "92878c5906f643a5aa9dafd7a1970f3f"
format_stamp: "Formatted at 2025-06-28 07:52:19 on dist-test-slave-wggc"
I20250628 07:52:19.528415 3943 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:19.596810 3943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:19.598296 3943 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:19.598713 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:19.601148 3943 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:19.605187 3943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:19.605384 3943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:19.605690 3943 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:19.605885 3943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:19.748807 3943 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.132:38513
I20250628 07:52:19.748919 4072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.132:38513 every 8 connection(s)
I20250628 07:52:19.751506 3943 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 07:52:19.759907 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 3943
I20250628 07:52:19.760643 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 07:52:19.774392 4073 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:19.774817 4073 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:19.775839 4073 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:19.778105 3485 ts_manager.cc:194] Registered new tserver with Master: 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513)
I20250628 07:52:19.779538 3485 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.132:41405
I20250628 07:52:19.782927 3438 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 07:52:19.826176 3485 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39360:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 07:52:19.907720 3608 tablet_service.cc:1468] Processing CreateTablet for tablet 1cece4233b544d559c926d467849e6fa (DEFAULT_TABLE table=TestTable [id=d2a583f35fb94740ba6dbbed3e51e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:19.907657 3741 tablet_service.cc:1468] Processing CreateTablet for tablet 1cece4233b544d559c926d467849e6fa (DEFAULT_TABLE table=TestTable [id=d2a583f35fb94740ba6dbbed3e51e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:19.909425 3741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cece4233b544d559c926d467849e6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:19.909478 3608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cece4233b544d559c926d467849e6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:19.919219 4008 tablet_service.cc:1468] Processing CreateTablet for tablet 1cece4233b544d559c926d467849e6fa (DEFAULT_TABLE table=TestTable [id=d2a583f35fb94740ba6dbbed3e51e022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:19.924228 4008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cece4233b544d559c926d467849e6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:19.963153 4092 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Bootstrap starting.
I20250628 07:52:19.964987 4093 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Bootstrap starting.
I20250628 07:52:19.968415 4094 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Bootstrap starting.
I20250628 07:52:19.972630 4092 tablet_bootstrap.cc:654] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:19.975198 4092 log.cc:826] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:19.975847 4094 tablet_bootstrap.cc:654] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:19.977135 4093 tablet_bootstrap.cc:654] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:19.978214 4094 log.cc:826] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:19.980185 4093 log.cc:826] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:19.988862 4093 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: No bootstrap required, opened a new log
I20250628 07:52:19.989547 4093 ts_tablet_manager.cc:1397] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.005s
I20250628 07:52:20.000129 4092 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: No bootstrap required, opened a new log
I20250628 07:52:20.000886 4092 ts_tablet_manager.cc:1397] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.009s
I20250628 07:52:20.004185 4094 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: No bootstrap required, opened a new log
I20250628 07:52:20.004812 4094 ts_tablet_manager.cc:1397] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.013s
I20250628 07:52:20.027140 4093 raft_consensus.cc:357] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.028228 4093 raft_consensus.cc:383] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:20.028587 4093 raft_consensus.cc:738] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362055de6a1245e0b5e27db34739dc33, State: Initialized, Role: FOLLOWER
I20250628 07:52:20.029559 4093 consensus_queue.cc:260] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.030313 4092 raft_consensus.cc:357] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.031605 4092 raft_consensus.cc:383] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:20.032056 4092 raft_consensus.cc:738] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319632b9d09f4ce9b2d72dda854b460f, State: Initialized, Role: FOLLOWER
I20250628 07:52:20.040903 4092 consensus_queue.cc:260] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.047853 4093 ts_tablet_manager.cc:1428] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Time spent starting tablet: real 0.058s user 0.042s sys 0.004s
I20250628 07:52:20.049239 4094 raft_consensus.cc:357] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.050284 4094 raft_consensus.cc:383] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:20.050354 4092 ts_tablet_manager.cc:1428] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Time spent starting tablet: real 0.049s user 0.034s sys 0.003s
I20250628 07:52:20.050683 4094 raft_consensus.cc:738] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92878c5906f643a5aa9dafd7a1970f3f, State: Initialized, Role: FOLLOWER
I20250628 07:52:20.051784 4094 consensus_queue.cc:260] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.055783 4073 heartbeater.cc:499] Master 127.3.91.190:43105 was elected leader, sending a full tablet report...
I20250628 07:52:20.056753 4094 ts_tablet_manager.cc:1428] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Time spent starting tablet: real 0.052s user 0.025s sys 0.007s
W20250628 07:52:20.102434 3674 tablet.cc:2378] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:52:20.247017 3807 tablet.cc:2378] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:52:20.256965 4074 tablet.cc:2378] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:20.333946 4100 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:52:20.334461 4100 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.337106 4100 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249), 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269)
I20250628 07:52:20.347501 3761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "92878c5906f643a5aa9dafd7a1970f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319632b9d09f4ce9b2d72dda854b460f" is_pre_election: true
I20250628 07:52:20.347884 3628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "92878c5906f643a5aa9dafd7a1970f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362055de6a1245e0b5e27db34739dc33" is_pre_election: true
I20250628 07:52:20.348292 3761 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92878c5906f643a5aa9dafd7a1970f3f in term 0.
I20250628 07:52:20.348649 3628 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92878c5906f643a5aa9dafd7a1970f3f in term 0.
I20250628 07:52:20.349638 3963 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 92878c5906f643a5aa9dafd7a1970f3f; no voters:
I20250628 07:52:20.350423 4100 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:52:20.350706 4100 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:52:20.350957 4100 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:20.355671 4100 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.357201 4100 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [CANDIDATE]: Term 1 election: Requested vote from peers 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249), 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269)
I20250628 07:52:20.358261 3761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "92878c5906f643a5aa9dafd7a1970f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "319632b9d09f4ce9b2d72dda854b460f"
I20250628 07:52:20.358392 3628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "92878c5906f643a5aa9dafd7a1970f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "362055de6a1245e0b5e27db34739dc33"
I20250628 07:52:20.358744 3761 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:20.358973 3628 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:20.363581 3761 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92878c5906f643a5aa9dafd7a1970f3f in term 1.
I20250628 07:52:20.364012 3628 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92878c5906f643a5aa9dafd7a1970f3f in term 1.
I20250628 07:52:20.364655 3963 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 92878c5906f643a5aa9dafd7a1970f3f; no voters:
I20250628 07:52:20.365695 4100 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:20.367841 4100 raft_consensus.cc:695] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [term 1 LEADER]: Becoming Leader. State: Replica: 92878c5906f643a5aa9dafd7a1970f3f, State: Running, Role: LEADER
I20250628 07:52:20.368917 4100 consensus_queue.cc:237] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:20.385993 3484 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132). New cstate: current_term: 1 leader_uuid: "92878c5906f643a5aa9dafd7a1970f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } health_report { overall_health: HEALTHY } } }
I20250628 07:52:20.482760 3438 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250628 07:52:20.487331 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 362055de6a1245e0b5e27db34739dc33 to finish bootstrapping
I20250628 07:52:20.501549 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 319632b9d09f4ce9b2d72dda854b460f to finish bootstrapping
I20250628 07:52:20.513513 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 92878c5906f643a5aa9dafd7a1970f3f to finish bootstrapping
I20250628 07:52:20.527048 3438 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250628 07:52:20.530601 3438 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "1cece4233b544d559c926d467849e6fa"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f"
rpc_addresses {
host: "127.3.91.130"
port: 39249
}
}
ts_infos {
permanent_uuid: "362055de6a1245e0b5e27db34739dc33"
rpc_addresses {
host: "127.3.91.129"
port: 34269
}
}
ts_infos {
permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f"
rpc_addresses {
host: "127.3.91.132"
port: 38513
}
}
I20250628 07:52:20.899746 4112 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:52:20.916412 4115 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 92878c5906f643a5aa9dafd7a1970f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:52:20.922178 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 3943
W20250628 07:52:20.956324 3562 connection.cc:537] server connection from 127.3.91.132:45541 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 07:52:20.957791 3470 connection.cc:537] server connection from 127.3.91.132:41405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250628 07:52:20.958755 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 3452
I20250628 07:52:20.985023 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43105
--webserver_interface=127.3.91.190
--webserver_port=35873
--builtin_ntp_servers=127.3.91.148:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:43105 with env {}
W20250628 07:52:21.305460 4122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:21.306142 4122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:21.306633 4122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:21.340003 4122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:52:21.340373 4122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:21.340641 4122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:52:21.340893 4122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:52:21.377066 4122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:43105
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=35873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:21.378540 4122 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:21.380347 4122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:21.395644 4129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:21.932559 3939 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:43105: connect: Connection refused (error 111)
W20250628 07:52:21.933705 3806 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:43105: connect: Connection refused (error 111)
W20250628 07:52:21.954119 3673 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:43105 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:43105: connect: Connection refused (error 111)
I20250628 07:52:22.414511 4137 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92878c5906f643a5aa9dafd7a1970f3f)
I20250628 07:52:22.414935 4137 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:22.417309 4137 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269), 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513)
W20250628 07:52:22.420624 3697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
W20250628 07:52:22.435701 3697 leader_election.cc:336] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513): Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
I20250628 07:52:22.436496 4141 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92878c5906f643a5aa9dafd7a1970f3f)
I20250628 07:52:22.437094 4141 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:22.441363 4141 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249), 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513)
W20250628 07:52:22.458357 3564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
I20250628 07:52:22.458165 3628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "319632b9d09f4ce9b2d72dda854b460f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "362055de6a1245e0b5e27db34739dc33" is_pre_election: true
I20250628 07:52:22.459398 3628 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 319632b9d09f4ce9b2d72dda854b460f in term 1.
I20250628 07:52:22.462308 3694 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 362055de6a1245e0b5e27db34739dc33; no voters: 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:22.463233 4137 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 07:52:22.463766 4137 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92878c5906f643a5aa9dafd7a1970f3f)
I20250628 07:52:22.464126 4137 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 1 FOLLOWER]: Advancing to term 2
W20250628 07:52:22.465709 3564 leader_election.cc:336] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513): Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
I20250628 07:52:22.468763 3761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "362055de6a1245e0b5e27db34739dc33" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "319632b9d09f4ce9b2d72dda854b460f" is_pre_election: true
I20250628 07:52:22.471728 4137 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:22.472862 3761 raft_consensus.cc:2391] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 362055de6a1245e0b5e27db34739dc33 in current term 2: Already voted for candidate 319632b9d09f4ce9b2d72dda854b460f in this term.
I20250628 07:52:22.474785 3628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "319632b9d09f4ce9b2d72dda854b460f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "362055de6a1245e0b5e27db34739dc33"
I20250628 07:52:22.475472 3628 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:52:22.476969 3563 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 362055de6a1245e0b5e27db34739dc33; no voters: 319632b9d09f4ce9b2d72dda854b460f, 92878c5906f643a5aa9dafd7a1970f3f
W20250628 07:52:22.478979 3697 leader_election.cc:336] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513): Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
I20250628 07:52:22.479636 4137 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 election: Requested vote from peers 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269), 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513)
I20250628 07:52:22.482057 3628 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 319632b9d09f4ce9b2d72dda854b460f in term 2.
I20250628 07:52:22.482622 4141 raft_consensus.cc:2747] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 07:52:22.483074 3694 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 362055de6a1245e0b5e27db34739dc33; no voters: 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:22.483738 4137 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:52:22.485914 4137 raft_consensus.cc:695] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 LEADER]: Becoming Leader. State: Replica: 319632b9d09f4ce9b2d72dda854b460f, State: Running, Role: LEADER
I20250628 07:52:22.486907 4137 consensus_queue.cc:237] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
W20250628 07:52:21.398969 4128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:22.629415 4130 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250628 07:52:22.630870 4131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:22.630916 4122 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:52:22.634860 4122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:22.637562 4122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:22.639117 4122 hybrid_clock.cc:648] HybridClock initialized: now 1751097142639081 us; error 40 us; skew 500 ppm
I20250628 07:52:22.640024 4122 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:22.651614 4122 webserver.cc:469] Webserver started at http://127.3.91.190:35873/ using document root <none> and password file <none>
I20250628 07:52:22.652660 4122 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:22.652979 4122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:22.661338 4122 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20250628 07:52:22.666375 4152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:22.667626 4122 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 07:52:22.667994 4122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "118fcb588aae44da8422e64b17075376"
format_stamp: "Formatted at 2025-06-28 07:52:11 on dist-test-slave-wggc"
I20250628 07:52:22.670061 4122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:22.738035 4122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:22.739578 4122 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:22.740026 4122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:22.815050 4122 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:43105
I20250628 07:52:22.815135 4203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:43105 every 8 connection(s)
I20250628 07:52:22.818063 4122 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:52:22.827271 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4122
I20250628 07:52:22.827767 3438 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 362055de6a1245e0b5e27db34739dc33
I20250628 07:52:22.828699 4204 sys_catalog.cc:263] Verifying existing consensus state
I20250628 07:52:22.833720 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Bootstrap starting.
I20250628 07:52:22.873850 4204 log.cc:826] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Log is configured to *not* fsync() on all Append() calls
W20250628 07:52:22.887977 3697 consensus_peers.cc:489] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f -> Peer 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513): Couldn't send request to peer 92878c5906f643a5aa9dafd7a1970f3f. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 07:52:22.906970 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:52:22.908111 4204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Bootstrap complete.
I20250628 07:52:22.945302 4204 raft_consensus.cc:357] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:22.949059 4204 raft_consensus.cc:738] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 118fcb588aae44da8422e64b17075376, State: Initialized, Role: FOLLOWER
I20250628 07:52:22.950240 4204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:22.951151 4204 raft_consensus.cc:397] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:52:22.951594 4204 raft_consensus.cc:491] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:52:22.952108 4204 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:52:22.963302 3939 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:22.963394 4204 raft_consensus.cc:513] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:22.964416 4204 leader_election.cc:304] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 118fcb588aae44da8422e64b17075376; no voters:
I20250628 07:52:22.967314 4204 leader_election.cc:290] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 07:52:22.968479 4211 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:52:22.970535 3628 raft_consensus.cc:1273] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 2 FOLLOWER]: Refusing update from remote peer 319632b9d09f4ce9b2d72dda854b460f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 07:52:22.973263 4137 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [LEADER]: Connected to new peer: Peer: permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250628 07:52:22.975091 4204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:52:22.979200 3673 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:22.987646 4211 raft_consensus.cc:695] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [term 2 LEADER]: Becoming Leader. State: Replica: 118fcb588aae44da8422e64b17075376, State: Running, Role: LEADER
I20250628 07:52:22.991751 4211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } }
I20250628 07:52:23.017355 4212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "118fcb588aae44da8422e64b17075376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } } }
I20250628 07:52:23.019181 4212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:23.020171 4213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 118fcb588aae44da8422e64b17075376. Latest consensus state: current_term: 2 leader_uuid: "118fcb588aae44da8422e64b17075376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "118fcb588aae44da8422e64b17075376" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43105 } } }
I20250628 07:52:23.022419 4213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:23.035776 4223 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:52:23.046557 3806 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43105
I20250628 07:52:23.057763 4223 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d2a583f35fb94740ba6dbbed3e51e022]
I20250628 07:52:23.078224 4223 tablet_loader.cc:96] loaded metadata for tablet 1cece4233b544d559c926d467849e6fa (table TestTable [id=d2a583f35fb94740ba6dbbed3e51e022])
I20250628 07:52:23.098995 4223 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:52:23.104310 4223 catalog_manager.cc:1261] Loaded cluster ID: 03bca85c0ee94834b0cca3a0fb9d34d6
I20250628 07:52:23.117367 4223 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:52:23.134891 4223 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:52:23.143123 4223 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 118fcb588aae44da8422e64b17075376: Loaded TSK: 0
I20250628 07:52:23.151125 4223 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 07:52:23.275322 4206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:23.276000 4206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:23.309957 4206 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 07:52:23.974784 4169 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" instance_seqno: 1751097137841580) as {username='slave'} at 127.3.91.131:34655; Asking this server to re-register.
I20250628 07:52:23.977198 3939 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:23.978133 3939 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:23.982026 4168 ts_manager.cc:194] Registered new tserver with Master: ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131:37951)
I20250628 07:52:24.027033 4169 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "362055de6a1245e0b5e27db34739dc33" instance_seqno: 1751097134054979) as {username='slave'} at 127.3.91.129:58205; Asking this server to re-register.
I20250628 07:52:24.029503 3673 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:24.030251 3673 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:24.033674 4169 ts_manager.cc:194] Registered new tserver with Master: 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269)
I20250628 07:52:24.038704 4169 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 reported cstate change: term changed from 1 to 2, leader changed from 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132) to 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130). New cstate: current_term: 2 leader_uuid: "319632b9d09f4ce9b2d72dda854b460f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } } }
I20250628 07:52:24.056365 4168 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" instance_seqno: 1751097135950385) as {username='slave'} at 127.3.91.130:48543; Asking this server to re-register.
I20250628 07:52:24.058897 3806 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:24.059823 3806 heartbeater.cc:507] Master 127.3.91.190:43105 requested a full tablet report, sending...
I20250628 07:52:24.064775 4168 ts_manager.cc:194] Registered new tserver with Master: 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249)
W20250628 07:52:24.762530 4238 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 4206
W20250628 07:52:24.763269 4238 kernel_stack_watchdog.cc:198] Thread 4206 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 07:52:24.785741 4206 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.428s user 0.509s sys 0.855s
W20250628 07:52:24.948731 4206 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.592s user 0.509s sys 0.871s
I20250628 07:52:24.998260 3628 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "362055de6a1245e0b5e27db34739dc33"
tablet_id: "1cece4233b544d559c926d467849e6fa"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f"
}
peers {
permanent_uuid: "362055de6a1245e0b5e27db34739dc33"
}
}
from {username='slave'} at 127.0.0.1:55784
W20250628 07:52:24.999712 3628 raft_consensus.cc:2216] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true
I20250628 07:52:25.000944 3628 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:52:25.160748 3628 raft_consensus.cc:1238] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Rejecting Update request from peer 319632b9d09f4ce9b2d72dda854b460f for earlier term 2. Current term is 3. Ops: []
I20250628 07:52:25.162459 4216 consensus_queue.cc:1046] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250628 07:52:25.164269 4216 raft_consensus.cc:3053] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 LEADER]: Stepping down as leader of term 2
I20250628 07:52:25.164572 4216 raft_consensus.cc:738] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 319632b9d09f4ce9b2d72dda854b460f, State: Running, Role: LEADER
I20250628 07:52:25.165246 4216 consensus_queue.cc:260] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:25.166265 4216 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:52:26.349381 4265 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:52:26.349813 4265 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } }
I20250628 07:52:26.351346 4265 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269), 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513)
I20250628 07:52:26.352265 3628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "319632b9d09f4ce9b2d72dda854b460f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "362055de6a1245e0b5e27db34739dc33" is_pre_election: true
W20250628 07:52:26.356067 3697 leader_election.cc:336] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132:38513): Network error: Client connection negotiation failed: client connection to 127.3.91.132:38513: connect: Connection refused (error 111)
I20250628 07:52:26.356396 3697 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f; no voters: 362055de6a1245e0b5e27db34739dc33, 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:26.356954 4265 raft_consensus.cc:2747] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250628 07:52:26.510368 4268 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250628 07:52:26.510779 4268 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true
I20250628 07:52:26.511869 4268 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249)
I20250628 07:52:26.513114 3761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "362055de6a1245e0b5e27db34739dc33" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "319632b9d09f4ce9b2d72dda854b460f" is_pre_election: true
I20250628 07:52:26.513664 3761 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362055de6a1245e0b5e27db34739dc33 in term 3.
I20250628 07:52:26.514612 3563 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 362055de6a1245e0b5e27db34739dc33; no voters:
I20250628 07:52:26.515226 4268 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250628 07:52:26.515480 4268 raft_consensus.cc:491] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250628 07:52:26.515729 4268 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 3 FOLLOWER]: Advancing to term 4
I20250628 07:52:26.519870 4268 raft_consensus.cc:513] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true
I20250628 07:52:26.520910 4268 leader_election.cc:290] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 4 election: Requested vote from peers 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130:39249)
I20250628 07:52:26.521962 3761 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1cece4233b544d559c926d467849e6fa" candidate_uuid: "362055de6a1245e0b5e27db34739dc33" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "319632b9d09f4ce9b2d72dda854b460f"
I20250628 07:52:26.522368 3761 raft_consensus.cc:3058] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 3 FOLLOWER]: Advancing to term 4
I20250628 07:52:26.526525 3761 raft_consensus.cc:2466] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362055de6a1245e0b5e27db34739dc33 in term 4.
I20250628 07:52:26.527367 3563 leader_election.cc:304] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 319632b9d09f4ce9b2d72dda854b460f, 362055de6a1245e0b5e27db34739dc33; no voters:
I20250628 07:52:26.527935 4268 raft_consensus.cc:2802] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 FOLLOWER]: Leader election won for term 4
I20250628 07:52:26.528733 4268 raft_consensus.cc:695] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 LEADER]: Becoming Leader. State: Replica: 362055de6a1245e0b5e27db34739dc33, State: Running, Role: LEADER
I20250628 07:52:26.529556 4268 consensus_queue.cc:237] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true
I20250628 07:52:26.535192 4169 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 reported cstate change: term changed from 2 to 4, leader changed from 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130) to 362055de6a1245e0b5e27db34739dc33 (127.3.91.129), now has a pending config: VOTER 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130), VOTER 362055de6a1245e0b5e27db34739dc33 (127.3.91.129). New cstate: current_term: 4 leader_uuid: "362055de6a1245e0b5e27db34739dc33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92878c5906f643a5aa9dafd7a1970f3f" member_type: VOTER last_known_addr { host: "127.3.91.132" port: 38513 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true }
I20250628 07:52:27.045046 3761 raft_consensus.cc:1273] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Refusing update from remote peer 362055de6a1245e0b5e27db34739dc33: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250628 07:52:27.046309 4275 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250628 07:52:27.054692 4278 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true }
I20250628 07:52:27.056018 3761 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } unsafe_config_change: true }
I20250628 07:52:27.065990 4169 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 reported cstate change: config changed from index -1 to 3, VOTER 92878c5906f643a5aa9dafd7a1970f3f (127.3.91.132) evicted, no longer has a pending config: VOTER 319632b9d09f4ce9b2d72dda854b460f (127.3.91.130), VOTER 362055de6a1245e0b5e27db34739dc33 (127.3.91.129). New cstate: current_term: 4 leader_uuid: "362055de6a1245e0b5e27db34739dc33" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250628 07:52:27.074635 4169 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 1cece4233b544d559c926d467849e6fa on TS 92878c5906f643a5aa9dafd7a1970f3f: Not found: failed to reset TS proxy: Could not find TS for UUID 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:27.093400 3628 consensus_queue.cc:237] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true
I20250628 07:52:27.098552 3761 raft_consensus.cc:1273] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Refusing update from remote peer 362055de6a1245e0b5e27db34739dc33: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250628 07:52:27.100069 4278 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250628 07:52:27.107646 4278 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true }
I20250628 07:52:27.109154 3761 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true }
W20250628 07:52:27.116647 3564 consensus_peers.cc:489] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 -> Peer ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131:37951): Couldn't send request to peer ddc625119ed24d268bd65a8422ffb6e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cece4233b544d559c926d467849e6fa. This is attempt 1: this message will repeat every 5th retry.
I20250628 07:52:27.119421 4153 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1cece4233b544d559c926d467849e6fa with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 07:52:27.121831 4168 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f reported cstate change: config changed from index 3 to 5, NON_VOTER ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) added. New cstate: current_term: 4 leader_uuid: "362055de6a1245e0b5e27db34739dc33" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true }
W20250628 07:52:27.127875 4154 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 1cece4233b544d559c926d467849e6fa on TS 92878c5906f643a5aa9dafd7a1970f3f failed: Not found: failed to reset TS proxy: Could not find TS for UUID 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:27.570406 4289 ts_tablet_manager.cc:927] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Initiating tablet copy from peer 362055de6a1245e0b5e27db34739dc33 (127.3.91.129:34269)
I20250628 07:52:27.573215 4289 tablet_copy_client.cc:323] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: tablet copy: Beginning tablet copy session from remote peer at address 127.3.91.129:34269
I20250628 07:52:27.584018 3648 tablet_copy_service.cc:140] P 362055de6a1245e0b5e27db34739dc33: Received BeginTabletCopySession request for tablet 1cece4233b544d559c926d467849e6fa from peer ddc625119ed24d268bd65a8422ffb6e4 ({username='slave'} at 127.3.91.131:45253)
I20250628 07:52:27.584507 3648 tablet_copy_service.cc:161] P 362055de6a1245e0b5e27db34739dc33: Beginning new tablet copy session on tablet 1cece4233b544d559c926d467849e6fa from peer ddc625119ed24d268bd65a8422ffb6e4 at {username='slave'} at 127.3.91.131:45253: session id = ddc625119ed24d268bd65a8422ffb6e4-1cece4233b544d559c926d467849e6fa
I20250628 07:52:27.589504 3648 tablet_copy_source_session.cc:215] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 07:52:27.594544 4289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cece4233b544d559c926d467849e6fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:27.613580 4289 tablet_copy_client.cc:806] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: tablet copy: Starting download of 0 data blocks...
I20250628 07:52:27.614215 4289 tablet_copy_client.cc:670] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: tablet copy: Starting download of 1 WAL segments...
I20250628 07:52:27.617949 4289 tablet_copy_client.cc:538] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 07:52:27.624523 4289 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Bootstrap starting.
I20250628 07:52:27.636847 4289 log.cc:826] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:27.647241 4289 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:52:27.648036 4289 tablet_bootstrap.cc:492] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Bootstrap complete.
I20250628 07:52:27.648650 4289 ts_tablet_manager.cc:1397] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.008s
I20250628 07:52:27.665560 4289 raft_consensus.cc:357] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true
I20250628 07:52:27.666487 4289 raft_consensus.cc:738] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: ddc625119ed24d268bd65a8422ffb6e4, State: Initialized, Role: LEARNER
I20250628 07:52:27.667124 4289 consensus_queue.cc:260] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: true } } unsafe_config_change: true
I20250628 07:52:27.670255 4289 ts_tablet_manager.cc:1428] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4: Time spent starting tablet: real 0.021s user 0.011s sys 0.013s
I20250628 07:52:27.671857 3648 tablet_copy_service.cc:342] P 362055de6a1245e0b5e27db34739dc33: Request end of tablet copy session ddc625119ed24d268bd65a8422ffb6e4-1cece4233b544d559c926d467849e6fa received from {username='slave'} at 127.3.91.131:45253
I20250628 07:52:27.672313 3648 tablet_copy_service.cc:434] P 362055de6a1245e0b5e27db34739dc33: ending tablet copy session ddc625119ed24d268bd65a8422ffb6e4-1cece4233b544d559c926d467849e6fa on tablet 1cece4233b544d559c926d467849e6fa with peer ddc625119ed24d268bd65a8422ffb6e4
I20250628 07:52:28.025204 3894 raft_consensus.cc:1215] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250628 07:52:28.295405 4154 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 1cece4233b544d559c926d467849e6fa on TS 92878c5906f643a5aa9dafd7a1970f3f failed: Not found: failed to reset TS proxy: Could not find TS for UUID 92878c5906f643a5aa9dafd7a1970f3f
I20250628 07:52:28.521626 4294 raft_consensus.cc:1062] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33: attempting to promote NON_VOTER ddc625119ed24d268bd65a8422ffb6e4 to VOTER
I20250628 07:52:28.523142 4294 consensus_queue.cc:237] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false } } unsafe_config_change: true
I20250628 07:52:28.528350 3761 raft_consensus.cc:1273] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Refusing update from remote peer 362055de6a1245e0b5e27db34739dc33: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250628 07:52:28.528424 3894 raft_consensus.cc:1273] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [term 4 LEARNER]: Refusing update from remote peer 362055de6a1245e0b5e27db34739dc33: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250628 07:52:28.529695 4297 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250628 07:52:28.530478 4295 consensus_queue.cc:1035] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250628 07:52:28.536231 4297 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false } } unsafe_config_change: true }
I20250628 07:52:28.538005 3761 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P 319632b9d09f4ce9b2d72dda854b460f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false } } unsafe_config_change: true }
I20250628 07:52:28.539084 3894 raft_consensus.cc:2953] T 1cece4233b544d559c926d467849e6fa P ddc625119ed24d268bd65a8422ffb6e4 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false } } unsafe_config_change: true }
I20250628 07:52:28.546422 4168 catalog_manager.cc:5582] T 1cece4233b544d559c926d467849e6fa P 362055de6a1245e0b5e27db34739dc33 reported cstate change: config changed from index 5 to 6, ddc625119ed24d268bd65a8422ffb6e4 (127.3.91.131) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "362055de6a1245e0b5e27db34739dc33" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "362055de6a1245e0b5e27db34739dc33" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 34269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 37951 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250628 07:52:28.560052 3438 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250628 07:52:28.583369 3438 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "1cece4233b544d559c926d467849e6fa"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "319632b9d09f4ce9b2d72dda854b460f"
rpc_addresses {
host: "127.3.91.130"
port: 39249
}
}
ts_infos {
permanent_uuid: "362055de6a1245e0b5e27db34739dc33"
rpc_addresses {
host: "127.3.91.129"
port: 34269
}
}
ts_infos {
permanent_uuid: "ddc625119ed24d268bd65a8422ffb6e4"
rpc_addresses {
host: "127.3.91.131"
port: 37951
}
}
I20250628 07:52:28.586905 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 3544
I20250628 07:52:28.612603 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 3677
I20250628 07:52:28.640710 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 3810
I20250628 07:52:28.664851 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4122
2025-06-28T07:52:28Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19931 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250628 07:52:28.724957 3438 test_util.cc:276] Using random seed: 327440067
I20250628 07:52:28.731106 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:52:28.731293 3438 ts_itest-base.cc:116] --------------
I20250628 07:52:28.731473 3438 ts_itest-base.cc:117] 3 tablet servers
I20250628 07:52:28.731616 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:52:28.731772 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:52:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:52:28Z Disabled control of system clock
I20250628 07:52:28.771010 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34221
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:34221
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250628 07:52:29.074144 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:29.074707 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:29.075114 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:29.106951 4315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:52:29.107259 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:29.107470 4315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:52:29.107666 4315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:52:29.144057 4315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:34221
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:29.145437 4315 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:29.147097 4315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:29.163507 4323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:29.163542 4325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:29.163545 4322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:29.165058 4315 server_base.cc:1048] running on GCE node
I20250628 07:52:30.349789 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:30.352579 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:30.354019 4315 hybrid_clock.cc:648] HybridClock initialized: now 1751097150353970 us; error 57 us; skew 500 ppm
I20250628 07:52:30.354880 4315 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:30.361303 4315 webserver.cc:469] Webserver started at http://127.3.91.190:38673/ using document root <none> and password file <none>
I20250628 07:52:30.362347 4315 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:30.362591 4315 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:30.363075 4315 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:30.367705 4315 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "20090442573146878a7080d5faef65e3"
format_stamp: "Formatted at 2025-06-28 07:52:30 on dist-test-slave-wggc"
I20250628 07:52:30.368863 4315 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "20090442573146878a7080d5faef65e3"
format_stamp: "Formatted at 2025-06-28 07:52:30 on dist-test-slave-wggc"
I20250628 07:52:30.376283 4315 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250628 07:52:30.382381 4332 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:30.383642 4315 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 07:52:30.384063 4315 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "20090442573146878a7080d5faef65e3"
format_stamp: "Formatted at 2025-06-28 07:52:30 on dist-test-slave-wggc"
I20250628 07:52:30.384387 4315 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:30.433658 4315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:30.435232 4315 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:30.435688 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:30.507436 4315 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:34221
I20250628 07:52:30.507524 4383 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:34221 every 8 connection(s)
I20250628 07:52:30.510172 4315 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:52:30.515303 4384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:30.517699 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4315
I20250628 07:52:30.518298 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:52:30.539984 4384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3: Bootstrap starting.
I20250628 07:52:30.546976 4384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:30.548662 4384 log.cc:826] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:30.553368 4384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3: No bootstrap required, opened a new log
I20250628 07:52:30.572044 4384 raft_consensus.cc:357] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } }
I20250628 07:52:30.573021 4384 raft_consensus.cc:383] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:30.573307 4384 raft_consensus.cc:738] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20090442573146878a7080d5faef65e3, State: Initialized, Role: FOLLOWER
I20250628 07:52:30.574178 4384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } }
I20250628 07:52:30.574743 4384 raft_consensus.cc:397] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:52:30.574983 4384 raft_consensus.cc:491] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:52:30.575258 4384 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:30.579308 4384 raft_consensus.cc:513] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } }
I20250628 07:52:30.580013 4384 leader_election.cc:304] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20090442573146878a7080d5faef65e3; no voters:
I20250628 07:52:30.582068 4384 leader_election.cc:290] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:52:30.582816 4389 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:30.585090 4389 raft_consensus.cc:695] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [term 1 LEADER]: Becoming Leader. State: Replica: 20090442573146878a7080d5faef65e3, State: Running, Role: LEADER
I20250628 07:52:30.585918 4389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } }
I20250628 07:52:30.586617 4384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:52:30.597257 4390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20090442573146878a7080d5faef65e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } } }
I20250628 07:52:30.598489 4390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:30.598193 4391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20090442573146878a7080d5faef65e3. Latest consensus state: current_term: 1 leader_uuid: "20090442573146878a7080d5faef65e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20090442573146878a7080d5faef65e3" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34221 } } }
I20250628 07:52:30.599716 4391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3 [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:30.602169 4398 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:52:30.614835 4398 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:52:30.632648 4398 catalog_manager.cc:1349] Generated new cluster ID: 62c0d711f89249218fe5d05410927251
I20250628 07:52:30.632987 4398 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:52:30.651126 4398 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:52:30.653206 4398 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:52:30.672323 4398 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 20090442573146878a7080d5faef65e3: Generated new TSK 0
I20250628 07:52:30.673497 4398 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:52:30.699275 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--builtin_ntp_servers=127.3.91.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250628 07:52:31.007242 4408 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 07:52:31.007934 4408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:31.008198 4408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:31.008704 4408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:31.040942 4408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:31.041949 4408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:52:31.078461 4408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:31.079818 4408 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:31.081477 4408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:31.099310 4417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:31.101788 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:31.099752 4415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:32.277511 4416 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:52:32.277700 4408 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:52:32.282632 4408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:32.285373 4408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:32.286875 4408 hybrid_clock.cc:648] HybridClock initialized: now 1751097152286823 us; error 95 us; skew 500 ppm
I20250628 07:52:32.287678 4408 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:32.294602 4408 webserver.cc:469] Webserver started at http://127.3.91.129:34181/ using document root <none> and password file <none>
I20250628 07:52:32.295547 4408 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:32.295765 4408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:32.296236 4408 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:32.300848 4408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f47bbd8784864bfbbfe0b467a1a99535"
format_stamp: "Formatted at 2025-06-28 07:52:32 on dist-test-slave-wggc"
I20250628 07:52:32.302110 4408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f47bbd8784864bfbbfe0b467a1a99535"
format_stamp: "Formatted at 2025-06-28 07:52:32 on dist-test-slave-wggc"
I20250628 07:52:32.309785 4408 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250628 07:52:32.315778 4424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:32.316870 4408 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250628 07:52:32.317198 4408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f47bbd8784864bfbbfe0b467a1a99535"
format_stamp: "Formatted at 2025-06-28 07:52:32 on dist-test-slave-wggc"
I20250628 07:52:32.317521 4408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:32.373584 4408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:32.375067 4408 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:32.375501 4408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:32.378005 4408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:32.382405 4408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:32.382622 4408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:32.382894 4408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:32.383049 4408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:32.519891 4408 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:35977
I20250628 07:52:32.519965 4536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:35977 every 8 connection(s)
I20250628 07:52:32.523448 4408 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:52:32.525573 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4408
I20250628 07:52:32.526165 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:52:32.533210 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--builtin_ntp_servers=127.3.91.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 07:52:32.546769 4537 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34221
I20250628 07:52:32.547215 4537 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:32.548247 4537 heartbeater.cc:507] Master 127.3.91.190:34221 requested a full tablet report, sending...
I20250628 07:52:32.550993 4349 ts_manager.cc:194] Registered new tserver with Master: f47bbd8784864bfbbfe0b467a1a99535 (127.3.91.129:35977)
I20250628 07:52:32.552943 4349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:37347
W20250628 07:52:32.837358 4541 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 07:52:32.838073 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:32.838344 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:32.838829 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:32.871053 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:32.871945 4541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:52:32.907805 4541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:32.909117 4541 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:32.910786 4541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:32.926940 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:33.556448 4537 heartbeater.cc:499] Master 127.3.91.190:34221 was elected leader, sending a full tablet report...
W20250628 07:52:32.927652 4550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:32.927966 4548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:32.931474 4541 server_base.cc:1048] running on GCE node
I20250628 07:52:34.102149 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:34.104622 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:34.105981 4541 hybrid_clock.cc:648] HybridClock initialized: now 1751097154105942 us; error 55 us; skew 500 ppm
I20250628 07:52:34.106784 4541 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:34.117173 4541 webserver.cc:469] Webserver started at http://127.3.91.130:43835/ using document root <none> and password file <none>
I20250628 07:52:34.118377 4541 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:34.118623 4541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:34.119149 4541 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:34.123785 4541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5"
format_stamp: "Formatted at 2025-06-28 07:52:34 on dist-test-slave-wggc"
I20250628 07:52:34.124980 4541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5"
format_stamp: "Formatted at 2025-06-28 07:52:34 on dist-test-slave-wggc"
I20250628 07:52:34.132669 4541 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250628 07:52:34.138355 4557 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:34.139407 4541 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250628 07:52:34.139742 4541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5"
format_stamp: "Formatted at 2025-06-28 07:52:34 on dist-test-slave-wggc"
I20250628 07:52:34.140110 4541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:34.203729 4541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:34.205237 4541 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:34.205703 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:34.208166 4541 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:34.212226 4541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:34.212453 4541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:34.212709 4541 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:34.212870 4541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:34.346302 4541 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:39335
I20250628 07:52:34.346421 4669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:39335 every 8 connection(s)
I20250628 07:52:34.348822 4541 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:52:34.350862 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4541
I20250628 07:52:34.351363 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:52:34.358175 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--builtin_ntp_servers=127.3.91.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250628 07:52:34.371186 4670 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34221
I20250628 07:52:34.371634 4670 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:34.372695 4670 heartbeater.cc:507] Master 127.3.91.190:34221 requested a full tablet report, sending...
I20250628 07:52:34.375217 4349 ts_manager.cc:194] Registered new tserver with Master: 1ff02c84aaa94e84bfdc6a81ad627ff5 (127.3.91.130:39335)
I20250628 07:52:34.376478 4349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:42547
W20250628 07:52:34.676936 4674 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250628 07:52:34.677583 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:34.677887 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:34.678376 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:34.709874 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:34.710738 4674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:52:34.745507 4674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:34.746937 4674 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:34.748589 4674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:34.763878 4680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:35.379662 4670 heartbeater.cc:499] Master 127.3.91.190:34221 was elected leader, sending a full tablet report...
W20250628 07:52:34.764501 4681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:34.765771 4683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:34.767390 4674 server_base.cc:1048] running on GCE node
I20250628 07:52:35.948743 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:35.951129 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:35.952504 4674 hybrid_clock.cc:648] HybridClock initialized: now 1751097155952465 us; error 60 us; skew 500 ppm
I20250628 07:52:35.953294 4674 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:35.960104 4674 webserver.cc:469] Webserver started at http://127.3.91.131:33013/ using document root <none> and password file <none>
I20250628 07:52:35.961108 4674 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:35.961382 4674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:35.961971 4674 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:35.966575 4674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "39756a76896d49daa09c880430f0bbe3"
format_stamp: "Formatted at 2025-06-28 07:52:35 on dist-test-slave-wggc"
I20250628 07:52:35.967657 4674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "39756a76896d49daa09c880430f0bbe3"
format_stamp: "Formatted at 2025-06-28 07:52:35 on dist-test-slave-wggc"
I20250628 07:52:35.975003 4674 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 07:52:35.980777 4690 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:35.981937 4674 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250628 07:52:35.982236 4674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "39756a76896d49daa09c880430f0bbe3"
format_stamp: "Formatted at 2025-06-28 07:52:35 on dist-test-slave-wggc"
I20250628 07:52:35.982558 4674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:36.031416 4674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:36.033082 4674 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:36.033499 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:36.036010 4674 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:36.040138 4674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:36.040350 4674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:36.040617 4674 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:36.040778 4674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:36.179802 4674 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:33427
I20250628 07:52:36.179908 4802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:33427 every 8 connection(s)
I20250628 07:52:36.182384 4674 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:52:36.186517 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4674
I20250628 07:52:36.186995 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:52:36.204058 4803 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34221
I20250628 07:52:36.204548 4803 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:36.205904 4803 heartbeater.cc:507] Master 127.3.91.190:34221 requested a full tablet report, sending...
I20250628 07:52:36.208096 4349 ts_manager.cc:194] Registered new tserver with Master: 39756a76896d49daa09c880430f0bbe3 (127.3.91.131:33427)
I20250628 07:52:36.209347 4349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:34403
I20250628 07:52:36.222062 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:52:36.258127 4349 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53620:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 07:52:36.277294 4349 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:52:36.325179 4605 tablet_service.cc:1468] Processing CreateTablet for tablet a335f2265a6e4f139145caef5addde22 (DEFAULT_TABLE table=TestTable [id=722a81ea2c174b6e9eef6adb6b2f9618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:36.325212 4472 tablet_service.cc:1468] Processing CreateTablet for tablet a335f2265a6e4f139145caef5addde22 (DEFAULT_TABLE table=TestTable [id=722a81ea2c174b6e9eef6adb6b2f9618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:36.327092 4472 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a335f2265a6e4f139145caef5addde22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:36.328066 4605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a335f2265a6e4f139145caef5addde22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:36.330729 4738 tablet_service.cc:1468] Processing CreateTablet for tablet a335f2265a6e4f139145caef5addde22 (DEFAULT_TABLE table=TestTable [id=722a81ea2c174b6e9eef6adb6b2f9618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:36.332700 4738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a335f2265a6e4f139145caef5addde22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:36.357138 4822 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Bootstrap starting.
I20250628 07:52:36.362689 4823 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Bootstrap starting.
I20250628 07:52:36.367340 4824 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Bootstrap starting.
I20250628 07:52:36.371400 4823 tablet_bootstrap.cc:654] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:36.374292 4822 tablet_bootstrap.cc:654] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:36.374455 4823 log.cc:826] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:36.377080 4822 log.cc:826] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:36.378064 4824 tablet_bootstrap.cc:654] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:36.381183 4824 log.cc:826] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:36.389451 4823 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: No bootstrap required, opened a new log
I20250628 07:52:36.390305 4823 ts_tablet_manager.cc:1397] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.011s
I20250628 07:52:36.398226 4822 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: No bootstrap required, opened a new log
I20250628 07:52:36.398712 4822 ts_tablet_manager.cc:1397] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.009s
I20250628 07:52:36.405843 4824 tablet_bootstrap.cc:492] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: No bootstrap required, opened a new log
I20250628 07:52:36.406421 4824 ts_tablet_manager.cc:1397] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.016s
I20250628 07:52:36.416244 4823 raft_consensus.cc:357] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.417007 4823 raft_consensus.cc:738] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff02c84aaa94e84bfdc6a81ad627ff5, State: Initialized, Role: FOLLOWER
I20250628 07:52:36.417798 4823 consensus_queue.cc:260] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.422318 4823 ts_tablet_manager.cc:1428] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Time spent starting tablet: real 0.032s user 0.029s sys 0.000s
I20250628 07:52:36.425951 4822 raft_consensus.cc:357] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.427101 4822 raft_consensus.cc:738] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f47bbd8784864bfbbfe0b467a1a99535, State: Initialized, Role: FOLLOWER
I20250628 07:52:36.427892 4822 consensus_queue.cc:260] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.431763 4822 ts_tablet_manager.cc:1428] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Time spent starting tablet: real 0.033s user 0.024s sys 0.008s
I20250628 07:52:36.434551 4824 raft_consensus.cc:357] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.435490 4824 raft_consensus.cc:738] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39756a76896d49daa09c880430f0bbe3, State: Initialized, Role: FOLLOWER
I20250628 07:52:36.436247 4824 consensus_queue.cc:260] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.441030 4803 heartbeater.cc:499] Master 127.3.91.190:34221 was elected leader, sending a full tablet report...
I20250628 07:52:36.442492 4824 ts_tablet_manager.cc:1428] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Time spent starting tablet: real 0.036s user 0.029s sys 0.005s
I20250628 07:52:36.460682 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:52:36.464150 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f47bbd8784864bfbbfe0b467a1a99535 to finish bootstrapping
I20250628 07:52:36.477679 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1ff02c84aaa94e84bfdc6a81ad627ff5 to finish bootstrapping
I20250628 07:52:36.488423 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 39756a76896d49daa09c880430f0bbe3 to finish bootstrapping
W20250628 07:52:36.532922 4538 tablet.cc:2378] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:36.539358 4492 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a335f2265a6e4f139145caef5addde22"
dest_uuid: "f47bbd8784864bfbbfe0b467a1a99535"
from {username='slave'} at 127.0.0.1:40872
I20250628 07:52:36.540082 4492 raft_consensus.cc:491] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 07:52:36.540467 4492 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:36.547282 4492 raft_consensus.cc:513] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.550364 4492 leader_election.cc:290] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [CANDIDATE]: Term 1 election: Requested vote from peers 1ff02c84aaa94e84bfdc6a81ad627ff5 (127.3.91.130:39335), 39756a76896d49daa09c880430f0bbe3 (127.3.91.131:33427)
I20250628 07:52:36.564757 3438 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250628 07:52:36.567668 4758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a335f2265a6e4f139145caef5addde22" candidate_uuid: "f47bbd8784864bfbbfe0b467a1a99535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39756a76896d49daa09c880430f0bbe3"
I20250628 07:52:36.568111 4625 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a335f2265a6e4f139145caef5addde22" candidate_uuid: "f47bbd8784864bfbbfe0b467a1a99535" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5"
I20250628 07:52:36.568567 4758 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:36.568866 4625 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:36.573637 4625 raft_consensus.cc:2466] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47bbd8784864bfbbfe0b467a1a99535 in term 1.
I20250628 07:52:36.574836 4428 leader_election.cc:304] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff02c84aaa94e84bfdc6a81ad627ff5, f47bbd8784864bfbbfe0b467a1a99535; no voters:
I20250628 07:52:36.574920 4758 raft_consensus.cc:2466] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f47bbd8784864bfbbfe0b467a1a99535 in term 1.
I20250628 07:52:36.575755 4829 raft_consensus.cc:2802] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:36.577394 4829 raft_consensus.cc:695] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 LEADER]: Becoming Leader. State: Replica: f47bbd8784864bfbbfe0b467a1a99535, State: Running, Role: LEADER
I20250628 07:52:36.578322 4829 consensus_queue.cc:237] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:36.589521 4348 catalog_manager.cc:5582] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 reported cstate change: term changed from 0 to 1, leader changed from <none> to f47bbd8784864bfbbfe0b467a1a99535 (127.3.91.129). New cstate: current_term: 1 leader_uuid: "f47bbd8784864bfbbfe0b467a1a99535" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } health_report { overall_health: UNKNOWN } } }
W20250628 07:52:36.606004 4671 tablet.cc:2378] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:36.670583 3438 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
W20250628 07:52:36.687093 4804 tablet.cc:2378] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:36.876164 3438 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250628 07:52:36.961755 4829 consensus_queue.cc:1035] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:52:36.977978 4840 consensus_queue.cc:1035] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:52:38.928894 4492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a335f2265a6e4f139145caef5addde22"
dest_uuid: "f47bbd8784864bfbbfe0b467a1a99535"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59544
I20250628 07:52:38.929431 4492 raft_consensus.cc:604] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 LEADER]: Received request to transfer leadership
I20250628 07:52:39.045362 4867 raft_consensus.cc:991] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535: : Instructing follower 1ff02c84aaa94e84bfdc6a81ad627ff5 to start an election
I20250628 07:52:39.045749 4867 raft_consensus.cc:1079] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 LEADER]: Signalling peer 1ff02c84aaa94e84bfdc6a81ad627ff5 to start an election
I20250628 07:52:39.047148 4625 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a335f2265a6e4f139145caef5addde22"
dest_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5"
from {username='slave'} at 127.3.91.129:44389
I20250628 07:52:39.047703 4625 raft_consensus.cc:491] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250628 07:52:39.047971 4625 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:52:39.053623 4625 raft_consensus.cc:513] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:39.057006 4625 leader_election.cc:290] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [CANDIDATE]: Term 2 election: Requested vote from peers f47bbd8784864bfbbfe0b467a1a99535 (127.3.91.129:35977), 39756a76896d49daa09c880430f0bbe3 (127.3.91.131:33427)
I20250628 07:52:39.071492 4492 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a335f2265a6e4f139145caef5addde22" candidate_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f47bbd8784864bfbbfe0b467a1a99535"
I20250628 07:52:39.072212 4492 raft_consensus.cc:3053] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 LEADER]: Stepping down as leader of term 1
I20250628 07:52:39.072583 4492 raft_consensus.cc:738] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f47bbd8784864bfbbfe0b467a1a99535, State: Running, Role: LEADER
I20250628 07:52:39.073206 4492 consensus_queue.cc:260] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:39.074239 4492 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:52:39.077036 4758 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a335f2265a6e4f139145caef5addde22" candidate_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39756a76896d49daa09c880430f0bbe3"
I20250628 07:52:39.077458 4758 raft_consensus.cc:3058] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:52:39.080010 4492 raft_consensus.cc:2466] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff02c84aaa94e84bfdc6a81ad627ff5 in term 2.
I20250628 07:52:39.081210 4560 leader_election.cc:304] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff02c84aaa94e84bfdc6a81ad627ff5, f47bbd8784864bfbbfe0b467a1a99535; no voters:
I20250628 07:52:39.081992 4758 raft_consensus.cc:2466] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ff02c84aaa94e84bfdc6a81ad627ff5 in term 2.
I20250628 07:52:39.083542 4872 raft_consensus.cc:2802] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:52:39.084928 4872 raft_consensus.cc:695] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [term 2 LEADER]: Becoming Leader. State: Replica: 1ff02c84aaa94e84bfdc6a81ad627ff5, State: Running, Role: LEADER
I20250628 07:52:39.085763 4872 consensus_queue.cc:237] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } }
I20250628 07:52:39.094559 4349 catalog_manager.cc:5582] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 reported cstate change: term changed from 1 to 2, leader changed from f47bbd8784864bfbbfe0b467a1a99535 (127.3.91.129) to 1ff02c84aaa94e84bfdc6a81ad627ff5 (127.3.91.130). New cstate: current_term: 2 leader_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ff02c84aaa94e84bfdc6a81ad627ff5" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 39335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 } health_report { overall_health: UNKNOWN } } }
I20250628 07:52:39.470994 4492 raft_consensus.cc:1273] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 2 FOLLOWER]: Refusing update from remote peer 1ff02c84aaa94e84bfdc6a81ad627ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 07:52:39.472349 4872 consensus_queue.cc:1035] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f47bbd8784864bfbbfe0b467a1a99535" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 35977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250628 07:52:39.482666 4758 raft_consensus.cc:1273] T a335f2265a6e4f139145caef5addde22 P 39756a76896d49daa09c880430f0bbe3 [term 2 FOLLOWER]: Refusing update from remote peer 1ff02c84aaa94e84bfdc6a81ad627ff5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250628 07:52:39.484710 4872 consensus_queue.cc:1035] T a335f2265a6e4f139145caef5addde22 P 1ff02c84aaa94e84bfdc6a81ad627ff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39756a76896d49daa09c880430f0bbe3" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250628 07:52:41.709051 4492 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a335f2265a6e4f139145caef5addde22"
dest_uuid: "f47bbd8784864bfbbfe0b467a1a99535"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59554
I20250628 07:52:41.709949 4492 raft_consensus.cc:604] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 2 FOLLOWER]: Received request to transfer leadership
I20250628 07:52:41.710404 4492 raft_consensus.cc:612] T a335f2265a6e4f139145caef5addde22 P f47bbd8784864bfbbfe0b467a1a99535 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250628 07:52:42.747808 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4408
I20250628 07:52:42.773443 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4541
I20250628 07:52:42.803506 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4674
I20250628 07:52:42.829644 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4315
2025-06-28T07:52:42Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14163 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250628 07:52:42.888506 3438 test_util.cc:276] Using random seed: 341603615
I20250628 07:52:42.892831 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:52:42.893009 3438 ts_itest-base.cc:116] --------------
I20250628 07:52:42.893183 3438 ts_itest-base.cc:117] 3 tablet servers
I20250628 07:52:42.893327 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:52:42.893476 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:52:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:52:42Z Disabled control of system clock
I20250628 07:52:42.931959 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:46295
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:37409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:46295 with env {}
W20250628 07:52:43.241355 4915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:43.242035 4915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:43.242516 4915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:43.274509 4915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:52:43.274904 4915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:43.275169 4915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:52:43.275442 4915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:52:43.312005 4915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:37409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:46295
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:43.313449 4915 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:43.315263 4915 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:43.332396 4922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:43.332415 4924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:43.334049 4915 server_base.cc:1048] running on GCE node
W20250628 07:52:43.332521 4921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:44.527794 4915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:44.530473 4915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:44.531898 4915 hybrid_clock.cc:648] HybridClock initialized: now 1751097164531847 us; error 67 us; skew 500 ppm
I20250628 07:52:44.532788 4915 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:44.539242 4915 webserver.cc:469] Webserver started at http://127.3.91.190:46315/ using document root <none> and password file <none>
I20250628 07:52:44.540261 4915 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:44.540510 4915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:44.541026 4915 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:44.545836 4915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "70764bce55984a009dbe4b81604d3edb"
format_stamp: "Formatted at 2025-06-28 07:52:44 on dist-test-slave-wggc"
I20250628 07:52:44.546988 4915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "70764bce55984a009dbe4b81604d3edb"
format_stamp: "Formatted at 2025-06-28 07:52:44 on dist-test-slave-wggc"
I20250628 07:52:44.554272 4915 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250628 07:52:44.559964 4931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:44.561034 4915 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250628 07:52:44.561379 4915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "70764bce55984a009dbe4b81604d3edb"
format_stamp: "Formatted at 2025-06-28 07:52:44 on dist-test-slave-wggc"
I20250628 07:52:44.561797 4915 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:44.619235 4915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:44.620788 4915 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:44.621258 4915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:44.695915 4915 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:46295
I20250628 07:52:44.695994 4982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:46295 every 8 connection(s)
I20250628 07:52:44.698690 4915 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:52:44.703712 4983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:44.707136 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 4915
I20250628 07:52:44.707675 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:52:44.727413 4983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb: Bootstrap starting.
I20250628 07:52:44.734171 4983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:44.735930 4983 log.cc:826] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:44.740660 4983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb: No bootstrap required, opened a new log
I20250628 07:52:44.761760 4983 raft_consensus.cc:357] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } }
I20250628 07:52:44.762535 4983 raft_consensus.cc:383] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:44.762858 4983 raft_consensus.cc:738] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70764bce55984a009dbe4b81604d3edb, State: Initialized, Role: FOLLOWER
I20250628 07:52:44.763834 4983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } }
I20250628 07:52:44.764416 4983 raft_consensus.cc:397] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:52:44.764755 4983 raft_consensus.cc:491] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:52:44.765102 4983 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:44.769500 4983 raft_consensus.cc:513] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } }
I20250628 07:52:44.770339 4983 leader_election.cc:304] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70764bce55984a009dbe4b81604d3edb; no voters:
I20250628 07:52:44.772297 4983 leader_election.cc:290] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:52:44.773168 4988 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:44.775432 4988 raft_consensus.cc:695] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [term 1 LEADER]: Becoming Leader. State: Replica: 70764bce55984a009dbe4b81604d3edb, State: Running, Role: LEADER
I20250628 07:52:44.776361 4988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } }
I20250628 07:52:44.776793 4983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:52:44.789712 4989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70764bce55984a009dbe4b81604d3edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } } }
I20250628 07:52:44.790759 4989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:44.794612 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70764bce55984a009dbe4b81604d3edb. Latest consensus state: current_term: 1 leader_uuid: "70764bce55984a009dbe4b81604d3edb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70764bce55984a009dbe4b81604d3edb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 46295 } } }
I20250628 07:52:44.795712 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb [sys.catalog]: This master's current role is: LEADER
I20250628 07:52:44.800400 4998 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:52:44.812587 4998 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:52:44.830418 4998 catalog_manager.cc:1349] Generated new cluster ID: aec5a4ffeac645e58d1b2f68ae0bb24e
I20250628 07:52:44.830749 4998 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:52:44.847985 4998 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:52:44.849646 4998 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:52:44.870924 4998 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 70764bce55984a009dbe4b81604d3edb: Generated new TSK 0
I20250628 07:52:44.871860 4998 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:52:44.889438 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--builtin_ntp_servers=127.3.91.148:37409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 07:52:45.224238 5007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:45.224829 5007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:45.225339 5007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:45.259389 5007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:45.260421 5007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:52:45.298213 5007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:37409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:45.299626 5007 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:45.301410 5007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:45.321128 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:45.324748 5007 server_base.cc:1048] running on GCE node
W20250628 07:52:45.325003 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:45.324643 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:46.556353 5007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:46.559279 5007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:46.560830 5007 hybrid_clock.cc:648] HybridClock initialized: now 1751097166560770 us; error 54 us; skew 500 ppm
I20250628 07:52:46.562050 5007 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:46.569877 5007 webserver.cc:469] Webserver started at http://127.3.91.129:33113/ using document root <none> and password file <none>
I20250628 07:52:46.571190 5007 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:46.571480 5007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:46.572069 5007 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:46.578855 5007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0c047f7f35ef4fec8559eec9095e00ca"
format_stamp: "Formatted at 2025-06-28 07:52:46 on dist-test-slave-wggc"
I20250628 07:52:46.580392 5007 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0c047f7f35ef4fec8559eec9095e00ca"
format_stamp: "Formatted at 2025-06-28 07:52:46 on dist-test-slave-wggc"
I20250628 07:52:46.589787 5007 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20250628 07:52:46.597668 5023 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:46.599031 5007 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:52:46.599429 5007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0c047f7f35ef4fec8559eec9095e00ca"
format_stamp: "Formatted at 2025-06-28 07:52:46 on dist-test-slave-wggc"
I20250628 07:52:46.599872 5007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:46.663935 5007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:46.666271 5007 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:46.666930 5007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:46.670692 5007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:46.676810 5007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:46.677175 5007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:46.677515 5007 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:46.677788 5007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.002s sys 0.000s
I20250628 07:52:46.821460 5007 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:40151
I20250628 07:52:46.821569 5135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:40151 every 8 connection(s)
I20250628 07:52:46.824131 5007 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:52:46.830235 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5007
I20250628 07:52:46.830794 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:52:46.836771 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--builtin_ntp_servers=127.3.91.148:37409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:52:46.850827 5136 heartbeater.cc:344] Connected to a master server at 127.3.91.190:46295
I20250628 07:52:46.851405 5136 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:46.852962 5136 heartbeater.cc:507] Master 127.3.91.190:46295 requested a full tablet report, sending...
I20250628 07:52:46.856171 4948 ts_manager.cc:194] Registered new tserver with Master: 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151)
I20250628 07:52:46.858170 4948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:45011
W20250628 07:52:47.147801 5140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:47.148350 5140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:47.148877 5140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:47.180665 5140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:47.181627 5140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:52:47.217408 5140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:37409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:47.218833 5140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:47.220573 5140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:47.236968 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:47.862253 5136 heartbeater.cc:499] Master 127.3.91.190:46295 was elected leader, sending a full tablet report...
W20250628 07:52:47.237094 5146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:47.238780 5149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:48.416546 5148 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:52:48.416585 5140 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:52:48.420684 5140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:48.423503 5140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:48.424989 5140 hybrid_clock.cc:648] HybridClock initialized: now 1751097168424942 us; error 61 us; skew 500 ppm
I20250628 07:52:48.425866 5140 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:48.432847 5140 webserver.cc:469] Webserver started at http://127.3.91.130:42709/ using document root <none> and password file <none>
I20250628 07:52:48.433889 5140 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:48.434098 5140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:48.434552 5140 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:48.439401 5140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7f9f3d51933a49a79035cf5613e4aea0"
format_stamp: "Formatted at 2025-06-28 07:52:48 on dist-test-slave-wggc"
I20250628 07:52:48.440521 5140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7f9f3d51933a49a79035cf5613e4aea0"
format_stamp: "Formatted at 2025-06-28 07:52:48 on dist-test-slave-wggc"
I20250628 07:52:48.448274 5140 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250628 07:52:48.454313 5156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:48.455447 5140 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 07:52:48.455775 5140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7f9f3d51933a49a79035cf5613e4aea0"
format_stamp: "Formatted at 2025-06-28 07:52:48 on dist-test-slave-wggc"
I20250628 07:52:48.456127 5140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:48.508842 5140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:48.510406 5140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:48.510838 5140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:48.513589 5140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:48.517817 5140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:48.518095 5140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:48.518308 5140 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:48.518453 5140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:48.657163 5140 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:45657
I20250628 07:52:48.657275 5268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:45657 every 8 connection(s)
I20250628 07:52:48.659804 5140 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:52:48.666404 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5140
I20250628 07:52:48.666844 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:52:48.672957 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--builtin_ntp_servers=127.3.91.148:37409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:52:48.683549 5269 heartbeater.cc:344] Connected to a master server at 127.3.91.190:46295
I20250628 07:52:48.684012 5269 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:48.685060 5269 heartbeater.cc:507] Master 127.3.91.190:46295 requested a full tablet report, sending...
I20250628 07:52:48.687534 4948 ts_manager.cc:194] Registered new tserver with Master: 7f9f3d51933a49a79035cf5613e4aea0 (127.3.91.130:45657)
I20250628 07:52:48.688822 4948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:50895
W20250628 07:52:49.001569 5273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:49.002146 5273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:52:49.002643 5273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:49.034056 5273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:52:49.034977 5273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:52:49.070773 5273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:37409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:46295
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:52:49.072199 5273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:52:49.073928 5273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:52:49.092787 5280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:49.692566 5269 heartbeater.cc:499] Master 127.3.91.190:46295 was elected leader, sending a full tablet report...
W20250628 07:52:49.092931 5281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:52:49.094127 5283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:52:49.095937 5273 server_base.cc:1048] running on GCE node
I20250628 07:52:50.278987 5273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:52:50.281244 5273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:52:50.282639 5273 hybrid_clock.cc:648] HybridClock initialized: now 1751097170282604 us; error 52 us; skew 500 ppm
I20250628 07:52:50.283485 5273 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:52:50.290433 5273 webserver.cc:469] Webserver started at http://127.3.91.131:46517/ using document root <none> and password file <none>
I20250628 07:52:50.291525 5273 fs_manager.cc:362] Metadata directory not provided
I20250628 07:52:50.291786 5273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:52:50.292367 5273 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:52:50.297515 5273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bd281c6bcd9246c19961a88d4ff144b9"
format_stamp: "Formatted at 2025-06-28 07:52:50 on dist-test-slave-wggc"
I20250628 07:52:50.298826 5273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bd281c6bcd9246c19961a88d4ff144b9"
format_stamp: "Formatted at 2025-06-28 07:52:50 on dist-test-slave-wggc"
I20250628 07:52:50.307206 5273 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250628 07:52:50.313540 5290 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:50.314837 5273 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:52:50.315201 5273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bd281c6bcd9246c19961a88d4ff144b9"
format_stamp: "Formatted at 2025-06-28 07:52:50 on dist-test-slave-wggc"
I20250628 07:52:50.315591 5273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:52:50.371724 5273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:52:50.373274 5273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:52:50.373869 5273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:52:50.376601 5273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:52:50.381148 5273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:52:50.381572 5273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:50.381894 5273 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:52:50.382063 5273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:52:50.524657 5273 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:41419
I20250628 07:52:50.524787 5402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:41419 every 8 connection(s)
I20250628 07:52:50.527427 5273 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:52:50.528302 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5273
I20250628 07:52:50.528875 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:52:50.550949 5403 heartbeater.cc:344] Connected to a master server at 127.3.91.190:46295
I20250628 07:52:50.551445 5403 heartbeater.cc:461] Registering TS with master...
I20250628 07:52:50.552609 5403 heartbeater.cc:507] Master 127.3.91.190:46295 requested a full tablet report, sending...
I20250628 07:52:50.554944 4948 ts_manager.cc:194] Registered new tserver with Master: bd281c6bcd9246c19961a88d4ff144b9 (127.3.91.131:41419)
I20250628 07:52:50.556242 4948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:60445
I20250628 07:52:50.566519 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:52:50.602500 4948 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54770:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 07:52:50.621729 4948 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:52:50.687772 5071 tablet_service.cc:1468] Processing CreateTablet for tablet 4a7f64fa86ed4097bd9adfe087394bcd (DEFAULT_TABLE table=TestTable [id=df59a10cdde847b3a68139e2430e8027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:50.687698 5204 tablet_service.cc:1468] Processing CreateTablet for tablet 4a7f64fa86ed4097bd9adfe087394bcd (DEFAULT_TABLE table=TestTable [id=df59a10cdde847b3a68139e2430e8027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:50.689956 5071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a7f64fa86ed4097bd9adfe087394bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:50.690634 5204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a7f64fa86ed4097bd9adfe087394bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:50.695897 5338 tablet_service.cc:1468] Processing CreateTablet for tablet 4a7f64fa86ed4097bd9adfe087394bcd (DEFAULT_TABLE table=TestTable [id=df59a10cdde847b3a68139e2430e8027]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:52:50.701038 5338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a7f64fa86ed4097bd9adfe087394bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:50.728842 5422 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Bootstrap starting.
I20250628 07:52:50.735903 5423 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Bootstrap starting.
I20250628 07:52:50.737964 5422 tablet_bootstrap.cc:654] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:50.741469 5422 log.cc:826] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:50.745357 5423 tablet_bootstrap.cc:654] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:50.745991 5424 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Bootstrap starting.
I20250628 07:52:50.748787 5423 log.cc:826] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:50.751546 5422 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: No bootstrap required, opened a new log
I20250628 07:52:50.752792 5422 ts_tablet_manager.cc:1397] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.007s
I20250628 07:52:50.754457 5424 tablet_bootstrap.cc:654] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:50.756949 5424 log.cc:826] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Log is configured to *not* fsync() on all Append() calls
I20250628 07:52:50.760362 5423 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: No bootstrap required, opened a new log
I20250628 07:52:50.760987 5423 ts_tablet_manager.cc:1397] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.007s
I20250628 07:52:50.781842 5424 tablet_bootstrap.cc:492] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: No bootstrap required, opened a new log
I20250628 07:52:50.782519 5424 ts_tablet_manager.cc:1397] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.007s
I20250628 07:52:50.785166 5422 raft_consensus.cc:357] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.786281 5422 raft_consensus.cc:383] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:50.786640 5422 raft_consensus.cc:738] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9f3d51933a49a79035cf5613e4aea0, State: Initialized, Role: FOLLOWER
I20250628 07:52:50.787597 5422 consensus_queue.cc:260] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.791797 5423 raft_consensus.cc:357] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.792834 5423 raft_consensus.cc:383] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:50.793180 5423 raft_consensus.cc:738] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd281c6bcd9246c19961a88d4ff144b9, State: Initialized, Role: FOLLOWER
I20250628 07:52:50.794368 5423 consensus_queue.cc:260] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.800685 5422 ts_tablet_manager.cc:1428] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Time spent starting tablet: real 0.048s user 0.026s sys 0.019s
I20250628 07:52:50.802642 5403 heartbeater.cc:499] Master 127.3.91.190:46295 was elected leader, sending a full tablet report...
I20250628 07:52:50.804936 5423 ts_tablet_manager.cc:1428] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Time spent starting tablet: real 0.044s user 0.029s sys 0.008s
I20250628 07:52:50.811337 5424 raft_consensus.cc:357] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.812389 5424 raft_consensus.cc:383] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:50.812677 5424 raft_consensus.cc:738] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c047f7f35ef4fec8559eec9095e00ca, State: Initialized, Role: FOLLOWER
I20250628 07:52:50.813477 5424 consensus_queue.cc:260] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:50.817395 5424 ts_tablet_manager.cc:1428] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Time spent starting tablet: real 0.035s user 0.029s sys 0.004s
W20250628 07:52:50.833423 5137 tablet.cc:2378] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:52:50.917063 5270 tablet.cc:2378] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:51.014145 5429 raft_consensus.cc:491] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:52:51.014667 5429 raft_consensus.cc:513] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.017020 5429 leader_election.cc:290] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f9f3d51933a49a79035cf5613e4aea0 (127.3.91.130:45657), 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151)
I20250628 07:52:51.025624 5224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a7f64fa86ed4097bd9adfe087394bcd" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9f3d51933a49a79035cf5613e4aea0" is_pre_election: true
I20250628 07:52:51.026651 5224 raft_consensus.cc:2466] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 0.
I20250628 07:52:51.028043 5293 leader_election.cc:304] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f9f3d51933a49a79035cf5613e4aea0, bd281c6bcd9246c19961a88d4ff144b9; no voters:
I20250628 07:52:51.028193 5091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a7f64fa86ed4097bd9adfe087394bcd" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c047f7f35ef4fec8559eec9095e00ca" is_pre_election: true
I20250628 07:52:51.028952 5429 raft_consensus.cc:2802] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:52:51.029059 5091 raft_consensus.cc:2466] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 0.
I20250628 07:52:51.029242 5429 raft_consensus.cc:491] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:52:51.029561 5429 raft_consensus.cc:3058] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Advancing to term 1
W20250628 07:52:51.033061 5404 tablet.cc:2378] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:52:51.034431 5429 raft_consensus.cc:513] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.035822 5429 leader_election.cc:290] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 election: Requested vote from peers 7f9f3d51933a49a79035cf5613e4aea0 (127.3.91.130:45657), 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151)
I20250628 07:52:51.036736 5224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a7f64fa86ed4097bd9adfe087394bcd" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9f3d51933a49a79035cf5613e4aea0"
I20250628 07:52:51.036864 5091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a7f64fa86ed4097bd9adfe087394bcd" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c047f7f35ef4fec8559eec9095e00ca"
I20250628 07:52:51.037168 5224 raft_consensus.cc:3058] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:51.037317 5091 raft_consensus.cc:3058] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:51.041741 5224 raft_consensus.cc:2466] T 4a7f64fa86ed4097bd9adfe087394bcd P 7f9f3d51933a49a79035cf5613e4aea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 1.
I20250628 07:52:51.041781 5091 raft_consensus.cc:2466] T 4a7f64fa86ed4097bd9adfe087394bcd P 0c047f7f35ef4fec8559eec9095e00ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 1.
I20250628 07:52:51.042732 5293 leader_election.cc:304] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f9f3d51933a49a79035cf5613e4aea0, bd281c6bcd9246c19961a88d4ff144b9; no voters:
I20250628 07:52:51.043418 5429 raft_consensus.cc:2802] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:51.044921 5429 raft_consensus.cc:695] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [term 1 LEADER]: Becoming Leader. State: Replica: bd281c6bcd9246c19961a88d4ff144b9, State: Running, Role: LEADER
I20250628 07:52:51.045732 5429 consensus_queue.cc:237] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.056758 4947 catalog_manager.cc:5582] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd281c6bcd9246c19961a88d4ff144b9 (127.3.91.131). New cstate: current_term: 1 leader_uuid: "bd281c6bcd9246c19961a88d4ff144b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } health_report { overall_health: HEALTHY } } }
I20250628 07:52:51.153111 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:52:51.156790 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0c047f7f35ef4fec8559eec9095e00ca to finish bootstrapping
I20250628 07:52:51.224165 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7f9f3d51933a49a79035cf5613e4aea0 to finish bootstrapping
I20250628 07:52:51.236609 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bd281c6bcd9246c19961a88d4ff144b9 to finish bootstrapping
I20250628 07:52:51.251368 4947 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54770:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250628 07:52:51.252974 4947 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:52:51.270146 5071 tablet_service.cc:1468] Processing CreateTablet for tablet 69f4498728be4fe289903d315b5ecf01 (DEFAULT_TABLE table=TestAnotherTable [id=1897367fb5014c58958afdf02264ac86]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 07:52:51.270710 5204 tablet_service.cc:1468] Processing CreateTablet for tablet 69f4498728be4fe289903d315b5ecf01 (DEFAULT_TABLE table=TestAnotherTable [id=1897367fb5014c58958afdf02264ac86]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 07:52:51.271322 5071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f4498728be4fe289903d315b5ecf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:51.271816 5204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f4498728be4fe289903d315b5ecf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:51.271541 5338 tablet_service.cc:1468] Processing CreateTablet for tablet 69f4498728be4fe289903d315b5ecf01 (DEFAULT_TABLE table=TestAnotherTable [id=1897367fb5014c58958afdf02264ac86]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250628 07:52:51.272516 5338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f4498728be4fe289903d315b5ecf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:52:51.284214 5423 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9: Bootstrap starting.
I20250628 07:52:51.286440 5424 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca: Bootstrap starting.
I20250628 07:52:51.287844 5422 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0: Bootstrap starting.
I20250628 07:52:51.289932 5423 tablet_bootstrap.cc:654] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:51.292888 5424 tablet_bootstrap.cc:654] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:51.293301 5422 tablet_bootstrap.cc:654] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0: Neither blocks nor log segments found. Creating new log.
I20250628 07:52:51.296947 5423 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9: No bootstrap required, opened a new log
I20250628 07:52:51.297472 5423 ts_tablet_manager.cc:1397] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20250628 07:52:51.298802 5424 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca: No bootstrap required, opened a new log
I20250628 07:52:51.299276 5424 ts_tablet_manager.cc:1397] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20250628 07:52:51.300921 5423 raft_consensus.cc:357] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.301496 5423 raft_consensus.cc:383] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:51.301872 5423 raft_consensus.cc:738] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd281c6bcd9246c19961a88d4ff144b9, State: Initialized, Role: FOLLOWER
I20250628 07:52:51.301983 5424 raft_consensus.cc:357] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.302597 5424 raft_consensus.cc:383] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:51.302810 5422 tablet_bootstrap.cc:492] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0: No bootstrap required, opened a new log
I20250628 07:52:51.302821 5424 raft_consensus.cc:738] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c047f7f35ef4fec8559eec9095e00ca, State: Initialized, Role: FOLLOWER
I20250628 07:52:51.302570 5423 consensus_queue.cc:260] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.303227 5422 ts_tablet_manager.cc:1397] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20250628 07:52:51.303431 5424 consensus_queue.cc:260] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.304790 5423 ts_tablet_manager.cc:1428] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250628 07:52:51.305188 5424 ts_tablet_manager.cc:1428] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250628 07:52:51.310221 5422 raft_consensus.cc:357] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.310957 5422 raft_consensus.cc:383] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:52:51.311240 5422 raft_consensus.cc:738] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f9f3d51933a49a79035cf5613e4aea0, State: Initialized, Role: FOLLOWER
I20250628 07:52:51.311913 5422 consensus_queue.cc:260] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.315356 5422 ts_tablet_manager.cc:1428] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250628 07:52:51.444154 5429 raft_consensus.cc:491] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:52:51.444612 5429 raft_consensus.cc:513] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.446170 5429 leader_election.cc:290] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151), 7f9f3d51933a49a79035cf5613e4aea0 (127.3.91.130:45657)
I20250628 07:52:51.447227 5091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f4498728be4fe289903d315b5ecf01" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c047f7f35ef4fec8559eec9095e00ca" is_pre_election: true
I20250628 07:52:51.447281 5224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f4498728be4fe289903d315b5ecf01" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9f3d51933a49a79035cf5613e4aea0" is_pre_election: true
I20250628 07:52:51.447737 5091 raft_consensus.cc:2466] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 0.
I20250628 07:52:51.447867 5224 raft_consensus.cc:2466] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 0.
I20250628 07:52:51.448753 5292 leader_election.cc:304] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c047f7f35ef4fec8559eec9095e00ca, bd281c6bcd9246c19961a88d4ff144b9; no voters:
I20250628 07:52:51.449558 5429 raft_consensus.cc:2802] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:52:51.449887 5429 raft_consensus.cc:491] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:52:51.450151 5429 raft_consensus.cc:3058] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:51.455235 5429 raft_consensus.cc:513] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.456784 5429 leader_election.cc:290] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 election: Requested vote from peers 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151), 7f9f3d51933a49a79035cf5613e4aea0 (127.3.91.130:45657)
I20250628 07:52:51.457763 5224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f4498728be4fe289903d315b5ecf01" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f9f3d51933a49a79035cf5613e4aea0"
I20250628 07:52:51.457818 5091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69f4498728be4fe289903d315b5ecf01" candidate_uuid: "bd281c6bcd9246c19961a88d4ff144b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c047f7f35ef4fec8559eec9095e00ca"
I20250628 07:52:51.458194 5224 raft_consensus.cc:3058] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:51.458268 5091 raft_consensus.cc:3058] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:52:51.462653 5091 raft_consensus.cc:2466] T 69f4498728be4fe289903d315b5ecf01 P 0c047f7f35ef4fec8559eec9095e00ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 1.
I20250628 07:52:51.462662 5224 raft_consensus.cc:2466] T 69f4498728be4fe289903d315b5ecf01 P 7f9f3d51933a49a79035cf5613e4aea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd281c6bcd9246c19961a88d4ff144b9 in term 1.
I20250628 07:52:51.463619 5292 leader_election.cc:304] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c047f7f35ef4fec8559eec9095e00ca, bd281c6bcd9246c19961a88d4ff144b9; no voters:
I20250628 07:52:51.464412 5429 raft_consensus.cc:2802] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:52:51.464809 5429 raft_consensus.cc:695] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [term 1 LEADER]: Becoming Leader. State: Replica: bd281c6bcd9246c19961a88d4ff144b9, State: Running, Role: LEADER
I20250628 07:52:51.465389 5429 consensus_queue.cc:237] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } }
I20250628 07:52:51.472541 4948 catalog_manager.cc:5582] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd281c6bcd9246c19961a88d4ff144b9 (127.3.91.131). New cstate: current_term: 1 leader_uuid: "bd281c6bcd9246c19961a88d4ff144b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd281c6bcd9246c19961a88d4ff144b9" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41419 } health_report { overall_health: HEALTHY } } }
I20250628 07:52:51.629725 5429 consensus_queue.cc:1035] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:52:51.649420 5432 consensus_queue.cc:1035] T 4a7f64fa86ed4097bd9adfe087394bcd P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 07:52:51.855012 5441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:51.855814 5441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:51.886904 5441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250628 07:52:51.951299 5446 consensus_queue.cc:1035] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c047f7f35ef4fec8559eec9095e00ca" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 07:52:51.975075 5446 consensus_queue.cc:1035] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f9f3d51933a49a79035cf5613e4aea0" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 07:52:52.214694 5132 debug-util.cc:398] Leaking SignalData structure 0x7b0800037740 after lost signal to thread 5008
W20250628 07:52:52.216207 5132 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7760 after lost signal to thread 5135
W20250628 07:52:53.231938 5441 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.303s user 0.476s sys 0.820s
W20250628 07:52:53.232353 5441 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.303s user 0.476s sys 0.820s
W20250628 07:52:54.653230 5464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:54.653935 5464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:54.704672 5464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 07:52:56.011580 5464 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.264s user 0.476s sys 0.786s
W20250628 07:52:56.012002 5464 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.264s user 0.476s sys 0.786s
W20250628 07:52:57.395816 5479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:52:57.396391 5479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:52:57.428620 5479 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 07:52:58.738379 5479 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.266s user 0.547s sys 0.718s
W20250628 07:52:58.738811 5479 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.266s user 0.547s sys 0.718s
W20250628 07:53:00.127877 5496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:00.128450 5496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:00.161284 5496 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 07:53:01.435614 5496 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.231s user 0.542s sys 0.686s
W20250628 07:53:01.436008 5496 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.231s user 0.542s sys 0.686s
I20250628 07:53:02.519411 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5007
I20250628 07:53:02.555502 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5140
W20250628 07:53:02.573168 5292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.129:40151: connect: Connection refused (error 111)
W20250628 07:53:02.577066 5292 consensus_peers.cc:489] T 69f4498728be4fe289903d315b5ecf01 P bd281c6bcd9246c19961a88d4ff144b9 -> Peer 0c047f7f35ef4fec8559eec9095e00ca (127.3.91.129:40151): Couldn't send request to peer 0c047f7f35ef4fec8559eec9095e00ca. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.129:40151: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 07:53:02.584020 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5273
I20250628 07:53:02.611075 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 4915
2025-06-28T07:53:02Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19784 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250628 07:53:02.672942 3438 test_util.cc:276] Using random seed: 361388050
I20250628 07:53:02.677233 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:53:02.677407 3438 ts_itest-base.cc:116] --------------
I20250628 07:53:02.677524 3438 ts_itest-base.cc:117] 3 tablet servers
I20250628 07:53:02.677690 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:53:02.677812 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:53:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:53:02Z Disabled control of system clock
I20250628 07:53:02.716025 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:35103
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:35103
--superuser_acl=no-such-user with env {}
W20250628 07:53:03.022163 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:03.022734 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:03.023183 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:03.055060 5518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:53:03.055384 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:03.055593 5518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:53:03.055788 5518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:53:03.092173 5518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:35103
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:35103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:03.093484 5518 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:03.095171 5518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:03.112188 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:03.112197 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:03.112371 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:03.113874 5518 server_base.cc:1048] running on GCE node
I20250628 07:53:04.299101 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:04.301923 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:04.303385 5518 hybrid_clock.cc:648] HybridClock initialized: now 1751097184303339 us; error 56 us; skew 500 ppm
I20250628 07:53:04.304267 5518 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:04.316418 5518 webserver.cc:469] Webserver started at http://127.3.91.190:34961/ using document root <none> and password file <none>
I20250628 07:53:04.317477 5518 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:04.317754 5518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:04.318251 5518 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:04.324717 5518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "16d1957018fb4c6ba725a6fa562bf6bf"
format_stamp: "Formatted at 2025-06-28 07:53:04 on dist-test-slave-wggc"
I20250628 07:53:04.326069 5518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "16d1957018fb4c6ba725a6fa562bf6bf"
format_stamp: "Formatted at 2025-06-28 07:53:04 on dist-test-slave-wggc"
I20250628 07:53:04.334344 5518 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20250628 07:53:04.340561 5534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:04.342001 5518 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250628 07:53:04.342410 5518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "16d1957018fb4c6ba725a6fa562bf6bf"
format_stamp: "Formatted at 2025-06-28 07:53:04 on dist-test-slave-wggc"
I20250628 07:53:04.342784 5518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:04.405828 5518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:04.407392 5518 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:04.407868 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:04.482213 5518 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:35103
I20250628 07:53:04.482288 5585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:35103 every 8 connection(s)
I20250628 07:53:04.484987 5518 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:53:04.490953 5586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:04.495409 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5518
I20250628 07:53:04.495846 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:53:04.514518 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf: Bootstrap starting.
I20250628 07:53:04.521091 5586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:04.523324 5586 log.cc:826] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:04.528718 5586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf: No bootstrap required, opened a new log
I20250628 07:53:04.548076 5586 raft_consensus.cc:357] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } }
I20250628 07:53:04.548784 5586 raft_consensus.cc:383] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:04.548988 5586 raft_consensus.cc:738] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d1957018fb4c6ba725a6fa562bf6bf, State: Initialized, Role: FOLLOWER
I20250628 07:53:04.549687 5586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } }
I20250628 07:53:04.550248 5586 raft_consensus.cc:397] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:04.550606 5586 raft_consensus.cc:491] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:04.551033 5586 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:04.556066 5586 raft_consensus.cc:513] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } }
I20250628 07:53:04.556810 5586 leader_election.cc:304] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d1957018fb4c6ba725a6fa562bf6bf; no voters:
I20250628 07:53:04.558533 5586 leader_election.cc:290] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:53:04.559232 5591 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:04.561664 5591 raft_consensus.cc:695] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [term 1 LEADER]: Becoming Leader. State: Replica: 16d1957018fb4c6ba725a6fa562bf6bf, State: Running, Role: LEADER
I20250628 07:53:04.562515 5591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } }
I20250628 07:53:04.562887 5586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:53:04.573980 5592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } } }
I20250628 07:53:04.574764 5592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:04.575701 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d1957018fb4c6ba725a6fa562bf6bf. Latest consensus state: current_term: 1 leader_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d1957018fb4c6ba725a6fa562bf6bf" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 35103 } } }
I20250628 07:53:04.576519 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:04.579180 5601 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:53:04.591069 5601 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:53:04.610095 5601 catalog_manager.cc:1349] Generated new cluster ID: bec6b1a297e843888d88a44b4e1ca316
I20250628 07:53:04.610524 5601 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:53:04.649204 5601 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:53:04.651444 5601 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:53:04.667510 5601 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 16d1957018fb4c6ba725a6fa562bf6bf: Generated new TSK 0
I20250628 07:53:04.668675 5601 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:53:04.694530 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--builtin_ntp_servers=127.3.91.148:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 07:53:05.047757 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:05.048324 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:05.048826 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:05.081539 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:05.082466 5610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:53:05.121150 5610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:05.122582 5610 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:05.124379 5610 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:05.142678 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:05.146260 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:05.143410 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:06.385759 5618 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1241 milliseconds
I20250628 07:53:06.385921 5610 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:06.387486 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:06.390389 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:06.391979 5610 hybrid_clock.cc:648] HybridClock initialized: now 1751097186391943 us; error 39 us; skew 500 ppm
I20250628 07:53:06.393126 5610 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:06.401527 5610 webserver.cc:469] Webserver started at http://127.3.91.129:43573/ using document root <none> and password file <none>
I20250628 07:53:06.402900 5610 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:06.403259 5610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:06.403947 5610 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:06.410723 5610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4da45dfd131242d397406346d7d8b5a5"
format_stamp: "Formatted at 2025-06-28 07:53:06 on dist-test-slave-wggc"
I20250628 07:53:06.412323 5610 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4da45dfd131242d397406346d7d8b5a5"
format_stamp: "Formatted at 2025-06-28 07:53:06 on dist-test-slave-wggc"
I20250628 07:53:06.422245 5610 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20250628 07:53:06.430128 5626 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:06.431447 5610 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250628 07:53:06.431941 5610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4da45dfd131242d397406346d7d8b5a5"
format_stamp: "Formatted at 2025-06-28 07:53:06 on dist-test-slave-wggc"
I20250628 07:53:06.432411 5610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:06.511611 5610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:06.513749 5610 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:06.514331 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:06.517781 5610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:06.523694 5610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:06.524004 5610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:06.524327 5610 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:06.524540 5610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:06.682425 5610 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:37843
I20250628 07:53:06.682502 5738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:37843 every 8 connection(s)
I20250628 07:53:06.686185 5610 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:53:06.694185 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5610
I20250628 07:53:06.694890 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:53:06.702641 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--builtin_ntp_servers=127.3.91.148:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:06.717881 5739 heartbeater.cc:344] Connected to a master server at 127.3.91.190:35103
I20250628 07:53:06.718508 5739 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:06.720089 5739 heartbeater.cc:507] Master 127.3.91.190:35103 requested a full tablet report, sending...
I20250628 07:53:06.723695 5551 ts_manager.cc:194] Registered new tserver with Master: 4da45dfd131242d397406346d7d8b5a5 (127.3.91.129:37843)
I20250628 07:53:06.726550 5551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:60197
W20250628 07:53:07.024562 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:07.025125 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:07.025717 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:07.058974 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:07.059973 5743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:53:07.097901 5743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:07.099304 5743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:07.101022 5743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:07.119776 5749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:07.732496 5739 heartbeater.cc:499] Master 127.3.91.190:35103 was elected leader, sending a full tablet report...
W20250628 07:53:07.119781 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:07.121359 5750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:08.318069 5751 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:08.318123 5743 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:08.321913 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:08.324152 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:08.325541 5743 hybrid_clock.cc:648] HybridClock initialized: now 1751097188325498 us; error 63 us; skew 500 ppm
I20250628 07:53:08.326426 5743 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:08.334054 5743 webserver.cc:469] Webserver started at http://127.3.91.130:35391/ using document root <none> and password file <none>
I20250628 07:53:08.335043 5743 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:08.335258 5743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:08.335739 5743 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:08.340502 5743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d6ac7c6fce744d2a883a1c639a5da3de"
format_stamp: "Formatted at 2025-06-28 07:53:08 on dist-test-slave-wggc"
I20250628 07:53:08.341787 5743 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d6ac7c6fce744d2a883a1c639a5da3de"
format_stamp: "Formatted at 2025-06-28 07:53:08 on dist-test-slave-wggc"
I20250628 07:53:08.349514 5743 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250628 07:53:08.356050 5759 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:08.357218 5743 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250628 07:53:08.357573 5743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d6ac7c6fce744d2a883a1c639a5da3de"
format_stamp: "Formatted at 2025-06-28 07:53:08 on dist-test-slave-wggc"
I20250628 07:53:08.357983 5743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:08.410005 5743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:08.411513 5743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:08.412014 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:08.414836 5743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:08.419090 5743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:08.419327 5743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:08.419613 5743 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:08.419775 5743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:08.566249 5743 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:40057
I20250628 07:53:08.566362 5871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:40057 every 8 connection(s)
I20250628 07:53:08.568930 5743 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:53:08.577414 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5743
I20250628 07:53:08.578052 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:53:08.584707 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--builtin_ntp_servers=127.3.91.148:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:08.593412 5872 heartbeater.cc:344] Connected to a master server at 127.3.91.190:35103
I20250628 07:53:08.593997 5872 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:08.595409 5872 heartbeater.cc:507] Master 127.3.91.190:35103 requested a full tablet report, sending...
I20250628 07:53:08.598086 5551 ts_manager.cc:194] Registered new tserver with Master: d6ac7c6fce744d2a883a1c639a5da3de (127.3.91.130:40057)
I20250628 07:53:08.599375 5551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:40455
W20250628 07:53:08.897094 5876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:08.897657 5876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:08.898175 5876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:08.932047 5876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:08.932947 5876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:53:08.968840 5876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:35103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:08.970261 5876 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:08.971979 5876 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:08.988767 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:09.602639 5872 heartbeater.cc:499] Master 127.3.91.190:35103 was elected leader, sending a full tablet report...
W20250628 07:53:08.988804 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:08.989795 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:10.186373 5885 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:10.186444 5876 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:10.191259 5876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:10.193744 5876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:10.195201 5876 hybrid_clock.cc:648] HybridClock initialized: now 1751097190195160 us; error 37 us; skew 500 ppm
I20250628 07:53:10.196064 5876 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:10.207857 5876 webserver.cc:469] Webserver started at http://127.3.91.131:34777/ using document root <none> and password file <none>
I20250628 07:53:10.208906 5876 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:10.209151 5876 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:10.209666 5876 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:10.214248 5876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3b351984a9ee4b8a93291047ab05bc66"
format_stamp: "Formatted at 2025-06-28 07:53:10 on dist-test-slave-wggc"
I20250628 07:53:10.215431 5876 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3b351984a9ee4b8a93291047ab05bc66"
format_stamp: "Formatted at 2025-06-28 07:53:10 on dist-test-slave-wggc"
I20250628 07:53:10.223891 5876 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20250628 07:53:10.230010 5893 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:10.231302 5876 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:53:10.231683 5876 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3b351984a9ee4b8a93291047ab05bc66"
format_stamp: "Formatted at 2025-06-28 07:53:10 on dist-test-slave-wggc"
I20250628 07:53:10.232046 5876 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:10.283058 5876 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:10.284606 5876 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:10.285097 5876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:10.287736 5876 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:10.292193 5876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:10.292465 5876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:10.292760 5876 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:10.292968 5876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:10.432837 5876 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:43741
I20250628 07:53:10.432922 6005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:43741 every 8 connection(s)
I20250628 07:53:10.435895 5876 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:53:10.446288 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 5876
I20250628 07:53:10.446873 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:53:10.458127 6006 heartbeater.cc:344] Connected to a master server at 127.3.91.190:35103
I20250628 07:53:10.458583 6006 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:10.459625 6006 heartbeater.cc:507] Master 127.3.91.190:35103 requested a full tablet report, sending...
I20250628 07:53:10.462035 5551 ts_manager.cc:194] Registered new tserver with Master: 3b351984a9ee4b8a93291047ab05bc66 (127.3.91.131:43741)
I20250628 07:53:10.463407 5551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:47225
I20250628 07:53:10.468182 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:10.507824 5551 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41492:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250628 07:53:10.527562 5551 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:53:10.593392 5941 tablet_service.cc:1468] Processing CreateTablet for tablet 6255f04289b440c68dad968f18217bdc (DEFAULT_TABLE table=TestTable [id=2f2be69c24b2417db400809f7e657166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:10.593744 5674 tablet_service.cc:1468] Processing CreateTablet for tablet 6255f04289b440c68dad968f18217bdc (DEFAULT_TABLE table=TestTable [id=2f2be69c24b2417db400809f7e657166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:10.593670 5807 tablet_service.cc:1468] Processing CreateTablet for tablet 6255f04289b440c68dad968f18217bdc (DEFAULT_TABLE table=TestTable [id=2f2be69c24b2417db400809f7e657166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:10.595253 5941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6255f04289b440c68dad968f18217bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:10.595556 5674 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6255f04289b440c68dad968f18217bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:10.597343 5807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6255f04289b440c68dad968f18217bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:10.617435 6025 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Bootstrap starting.
I20250628 07:53:10.624305 6026 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Bootstrap starting.
I20250628 07:53:10.625356 6025 tablet_bootstrap.cc:654] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:10.627851 6025 log.cc:826] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:10.629768 6027 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Bootstrap starting.
I20250628 07:53:10.631275 6026 tablet_bootstrap.cc:654] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:10.634166 6025 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: No bootstrap required, opened a new log
I20250628 07:53:10.634676 6025 ts_tablet_manager.cc:1397] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20250628 07:53:10.635743 6026 log.cc:826] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:10.637907 6027 tablet_bootstrap.cc:654] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:10.640234 6027 log.cc:826] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:10.641858 6026 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: No bootstrap required, opened a new log
I20250628 07:53:10.642601 6026 ts_tablet_manager.cc:1397] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250628 07:53:10.646282 6027 tablet_bootstrap.cc:492] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: No bootstrap required, opened a new log
I20250628 07:53:10.646852 6027 ts_tablet_manager.cc:1397] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20250628 07:53:10.655405 6025 raft_consensus.cc:357] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.656174 6025 raft_consensus.cc:383] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:10.656612 6025 raft_consensus.cc:738] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b351984a9ee4b8a93291047ab05bc66, State: Initialized, Role: FOLLOWER
I20250628 07:53:10.657768 6025 consensus_queue.cc:260] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.661055 6006 heartbeater.cc:499] Master 127.3.91.190:35103 was elected leader, sending a full tablet report...
I20250628 07:53:10.662568 6025 ts_tablet_manager.cc:1428] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Time spent starting tablet: real 0.028s user 0.025s sys 0.002s
I20250628 07:53:10.672569 6026 raft_consensus.cc:357] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.673741 6026 raft_consensus.cc:383] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:10.674101 6026 raft_consensus.cc:738] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da45dfd131242d397406346d7d8b5a5, State: Initialized, Role: FOLLOWER
I20250628 07:53:10.675105 6026 consensus_queue.cc:260] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.675215 6027 raft_consensus.cc:357] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.676141 6027 raft_consensus.cc:383] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:10.676477 6027 raft_consensus.cc:738] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ac7c6fce744d2a883a1c639a5da3de, State: Initialized, Role: FOLLOWER
I20250628 07:53:10.677675 6027 consensus_queue.cc:260] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.679657 6026 ts_tablet_manager.cc:1428] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Time spent starting tablet: real 0.037s user 0.029s sys 0.004s
I20250628 07:53:10.682936 6027 ts_tablet_manager.cc:1428] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Time spent starting tablet: real 0.036s user 0.031s sys 0.005s
W20250628 07:53:10.690829 6007 tablet.cc:2378] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:53:10.695711 5740 tablet.cc:2378] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:53:10.786373 6031 raft_consensus.cc:491] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:53:10.786875 6031 raft_consensus.cc:513] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.789211 6031 leader_election.cc:290] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4da45dfd131242d397406346d7d8b5a5 (127.3.91.129:37843), d6ac7c6fce744d2a883a1c639a5da3de (127.3.91.130:40057)
I20250628 07:53:10.798382 5694 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6255f04289b440c68dad968f18217bdc" candidate_uuid: "3b351984a9ee4b8a93291047ab05bc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da45dfd131242d397406346d7d8b5a5" is_pre_election: true
I20250628 07:53:10.799363 5694 raft_consensus.cc:2466] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b351984a9ee4b8a93291047ab05bc66 in term 0.
I20250628 07:53:10.800639 5894 leader_election.cc:304] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b351984a9ee4b8a93291047ab05bc66, 4da45dfd131242d397406346d7d8b5a5; no voters:
I20250628 07:53:10.801095 5827 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6255f04289b440c68dad968f18217bdc" candidate_uuid: "3b351984a9ee4b8a93291047ab05bc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" is_pre_election: true
I20250628 07:53:10.801401 6031 raft_consensus.cc:2802] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:53:10.801786 6031 raft_consensus.cc:491] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:53:10.802114 6031 raft_consensus.cc:3058] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:10.802021 5827 raft_consensus.cc:2466] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b351984a9ee4b8a93291047ab05bc66 in term 0.
I20250628 07:53:10.806790 6031 raft_consensus.cc:513] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
I20250628 07:53:10.808238 6031 leader_election.cc:290] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [CANDIDATE]: Term 1 election: Requested vote from peers 4da45dfd131242d397406346d7d8b5a5 (127.3.91.129:37843), d6ac7c6fce744d2a883a1c639a5da3de (127.3.91.130:40057)
I20250628 07:53:10.809161 5694 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6255f04289b440c68dad968f18217bdc" candidate_uuid: "3b351984a9ee4b8a93291047ab05bc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4da45dfd131242d397406346d7d8b5a5"
I20250628 07:53:10.809262 5827 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6255f04289b440c68dad968f18217bdc" candidate_uuid: "3b351984a9ee4b8a93291047ab05bc66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6ac7c6fce744d2a883a1c639a5da3de"
I20250628 07:53:10.809744 5827 raft_consensus.cc:3058] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:10.809808 5694 raft_consensus.cc:3058] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:10.814540 5827 raft_consensus.cc:2466] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b351984a9ee4b8a93291047ab05bc66 in term 1.
I20250628 07:53:10.814594 5694 raft_consensus.cc:2466] T 6255f04289b440c68dad968f18217bdc P 4da45dfd131242d397406346d7d8b5a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b351984a9ee4b8a93291047ab05bc66 in term 1.
I20250628 07:53:10.815554 5894 leader_election.cc:304] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b351984a9ee4b8a93291047ab05bc66, d6ac7c6fce744d2a883a1c639a5da3de; no voters:
I20250628 07:53:10.816385 6031 raft_consensus.cc:2802] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:10.817981 6031 raft_consensus.cc:695] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [term 1 LEADER]: Becoming Leader. State: Replica: 3b351984a9ee4b8a93291047ab05bc66, State: Running, Role: LEADER
I20250628 07:53:10.818861 6031 consensus_queue.cc:237] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } }
W20250628 07:53:10.826140 5873 tablet.cc:2378] T 6255f04289b440c68dad968f18217bdc P d6ac7c6fce744d2a883a1c639a5da3de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:53:10.829764 5551 catalog_manager.cc:5582] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b351984a9ee4b8a93291047ab05bc66 (127.3.91.131). New cstate: current_term: 1 leader_uuid: "3b351984a9ee4b8a93291047ab05bc66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b351984a9ee4b8a93291047ab05bc66" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 43741 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:10.883344 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:10.886917 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4da45dfd131242d397406346d7d8b5a5 to finish bootstrapping
I20250628 07:53:10.899912 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d6ac7c6fce744d2a883a1c639a5da3de to finish bootstrapping
I20250628 07:53:10.911309 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3b351984a9ee4b8a93291047ab05bc66 to finish bootstrapping
I20250628 07:53:11.359500 6048 consensus_queue.cc:1035] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ac7c6fce744d2a883a1c639a5da3de" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:53:11.398576 6054 consensus_queue.cc:1035] T 6255f04289b440c68dad968f18217bdc P 3b351984a9ee4b8a93291047ab05bc66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da45dfd131242d397406346d7d8b5a5" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250628 07:53:12.641655 5551 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:41506
I20250628 07:53:13.671993 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5610
I20250628 07:53:13.698896 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5743
I20250628 07:53:13.725432 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5876
I20250628 07:53:13.751718 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 5518
2025-06-28T07:53:13Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11132 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250628 07:53:13.805586 3438 test_util.cc:276] Using random seed: 372520686
I20250628 07:53:13.809832 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:53:13.810004 3438 ts_itest-base.cc:116] --------------
I20250628 07:53:13.810124 3438 ts_itest-base.cc:117] 3 tablet servers
I20250628 07:53:13.810235 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:53:13.810350 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:53:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:53:13Z Disabled control of system clock
I20250628 07:53:13.848311 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:42237 with env {}
W20250628 07:53:14.165939 6075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:14.166513 6075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:14.166927 6075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:14.198760 6075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:53:14.199069 6075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:14.199275 6075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:53:14.199479 6075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:53:14.236267 6075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:42237
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:14.237709 6075 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:14.239346 6075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:14.256347 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:14.256393 6082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:14.256421 6081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:14.257705 6075 server_base.cc:1048] running on GCE node
I20250628 07:53:15.447916 6075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:15.450711 6075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:15.452100 6075 hybrid_clock.cc:648] HybridClock initialized: now 1751097195452053 us; error 55 us; skew 500 ppm
I20250628 07:53:15.452987 6075 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:15.465260 6075 webserver.cc:469] Webserver started at http://127.3.91.190:35707/ using document root <none> and password file <none>
I20250628 07:53:15.466292 6075 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:15.466522 6075 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:15.467021 6075 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:15.471679 6075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:15.472847 6075 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:15.480247 6075 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250628 07:53:15.486716 6091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:15.487977 6075 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250628 07:53:15.488324 6075 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:15.488678 6075 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:15.550758 6075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:15.552203 6075 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:15.552615 6075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:15.622501 6075 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:42237
I20250628 07:53:15.622761 6142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:42237 every 8 connection(s)
I20250628 07:53:15.625247 6075 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:53:15.630738 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6075
I20250628 07:53:15.630844 6143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:15.631263 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:53:15.655970 6143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap starting.
I20250628 07:53:15.661772 6143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:15.663527 6143 log.cc:826] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:15.667753 6143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: No bootstrap required, opened a new log
I20250628 07:53:15.684391 6143 raft_consensus.cc:357] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:15.685052 6143 raft_consensus.cc:383] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:15.685302 6143 raft_consensus.cc:738] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Initialized, Role: FOLLOWER
I20250628 07:53:15.686061 6143 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:15.686560 6143 raft_consensus.cc:397] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:15.686822 6143 raft_consensus.cc:491] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:15.687141 6143 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:15.691344 6143 raft_consensus.cc:513] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:15.692067 6143 leader_election.cc:304] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a19ec6f3d445709d6936fbffe0e1fb; no voters:
I20250628 07:53:15.693768 6143 leader_election.cc:290] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:53:15.694442 6148 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:15.696623 6148 raft_consensus.cc:695] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 1 LEADER]: Becoming Leader. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Running, Role: LEADER
I20250628 07:53:15.697372 6148 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:15.698376 6143 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:53:15.708104 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a19ec6f3d445709d6936fbffe0e1fb. Latest consensus state: current_term: 1 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:15.709651 6149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:15.710310 6149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:15.709980 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:15.716678 6158 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:53:15.729770 6158 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:53:15.747332 6158 catalog_manager.cc:1349] Generated new cluster ID: 4e3ba728b1464b929e513a3c12c5c940
I20250628 07:53:15.747627 6158 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:53:15.780265 6158 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:53:15.781921 6158 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:53:15.796995 6158 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Generated new TSK 0
I20250628 07:53:15.798027 6158 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:53:15.819061 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 07:53:16.130192 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:16.130728 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:16.131225 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:16.163851 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:16.164724 6167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:53:16.199826 6167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:16.201153 6167 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:16.202886 6167 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:16.222983 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:16.224910 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:16.224556 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:16.226642 6167 server_base.cc:1048] running on GCE node
I20250628 07:53:17.421371 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:17.424311 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:17.425720 6167 hybrid_clock.cc:648] HybridClock initialized: now 1751097197425655 us; error 84 us; skew 500 ppm
I20250628 07:53:17.426530 6167 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:17.434077 6167 webserver.cc:469] Webserver started at http://127.3.91.129:46259/ using document root <none> and password file <none>
I20250628 07:53:17.435060 6167 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:17.435282 6167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:17.435750 6167 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:17.440223 6167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
format_stamp: "Formatted at 2025-06-28 07:53:17 on dist-test-slave-wggc"
I20250628 07:53:17.441331 6167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
format_stamp: "Formatted at 2025-06-28 07:53:17 on dist-test-slave-wggc"
I20250628 07:53:17.448901 6167 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250628 07:53:17.455226 6183 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:17.456341 6167 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250628 07:53:17.456658 6167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
format_stamp: "Formatted at 2025-06-28 07:53:17 on dist-test-slave-wggc"
I20250628 07:53:17.456974 6167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:17.510872 6167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:17.512432 6167 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:17.512938 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:17.515645 6167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:17.520385 6167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:17.520599 6167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:17.520862 6167 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:17.521023 6167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:17.660530 6167 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:44329
I20250628 07:53:17.660636 6295 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:44329 every 8 connection(s)
I20250628 07:53:17.663072 6167 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:53:17.672215 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6167
I20250628 07:53:17.672622 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:53:17.679122 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:17.685315 6296 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:17.685791 6296 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:17.686822 6296 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:17.689765 6108 ts_manager.cc:194] Registered new tserver with Master: 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329)
I20250628 07:53:17.691834 6108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:37121
W20250628 07:53:17.987260 6300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:17.987762 6300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:17.988411 6300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:18.021458 6300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:18.022578 6300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:53:18.059731 6300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:18.061126 6300 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:18.062902 6300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:18.079928 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:18.695917 6296 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
W20250628 07:53:18.080343 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:18.083348 6300 server_base.cc:1048] running on GCE node
W20250628 07:53:18.080641 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:19.256911 6300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:19.259809 6300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:19.261333 6300 hybrid_clock.cc:648] HybridClock initialized: now 1751097199261283 us; error 64 us; skew 500 ppm
I20250628 07:53:19.262256 6300 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:19.269407 6300 webserver.cc:469] Webserver started at http://127.3.91.130:37845/ using document root <none> and password file <none>
I20250628 07:53:19.270460 6300 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:19.270666 6300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:19.271061 6300 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:19.275763 6300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ce5b97082f844f69a92cb49f0407e949"
format_stamp: "Formatted at 2025-06-28 07:53:19 on dist-test-slave-wggc"
I20250628 07:53:19.276894 6300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ce5b97082f844f69a92cb49f0407e949"
format_stamp: "Formatted at 2025-06-28 07:53:19 on dist-test-slave-wggc"
I20250628 07:53:19.284715 6300 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20250628 07:53:19.291055 6317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:19.292212 6300 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250628 07:53:19.292565 6300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce5b97082f844f69a92cb49f0407e949"
format_stamp: "Formatted at 2025-06-28 07:53:19 on dist-test-slave-wggc"
I20250628 07:53:19.292902 6300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:19.354712 6300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:19.356231 6300 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:19.356680 6300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:19.359254 6300 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:19.363435 6300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:19.363667 6300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:19.363945 6300 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:19.364107 6300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:19.502380 6300 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:33649
I20250628 07:53:19.502482 6429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:33649 every 8 connection(s)
I20250628 07:53:19.504957 6300 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:53:19.511631 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6300
I20250628 07:53:19.512351 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:53:19.519629 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:19.527966 6430 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:19.528462 6430 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:19.529500 6430 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:19.531911 6108 ts_manager.cc:194] Registered new tserver with Master: ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649)
I20250628 07:53:19.533823 6108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:60059
W20250628 07:53:19.828995 6434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:19.829474 6434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:19.829988 6434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:19.862026 6434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:19.862864 6434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:53:19.898212 6434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:19.899544 6434 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:19.901209 6434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:19.916867 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:20.537392 6430 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
W20250628 07:53:19.919353 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:19.917017 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:21.131168 6442 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1211 milliseconds
I20250628 07:53:21.131292 6434 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:21.132495 6434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:21.135298 6434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:21.136830 6434 hybrid_clock.cc:648] HybridClock initialized: now 1751097201136779 us; error 63 us; skew 500 ppm
I20250628 07:53:21.137733 6434 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:21.149351 6434 webserver.cc:469] Webserver started at http://127.3.91.131:46791/ using document root <none> and password file <none>
I20250628 07:53:21.150403 6434 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:21.150615 6434 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:21.151016 6434 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:53:21.157497 6434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "84598f3817904a8794eebb4582ab6494"
format_stamp: "Formatted at 2025-06-28 07:53:21 on dist-test-slave-wggc"
I20250628 07:53:21.158730 6434 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "84598f3817904a8794eebb4582ab6494"
format_stamp: "Formatted at 2025-06-28 07:53:21 on dist-test-slave-wggc"
I20250628 07:53:21.166026 6434 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250628 07:53:21.171886 6450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:21.173002 6434 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250628 07:53:21.173318 6434 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "84598f3817904a8794eebb4582ab6494"
format_stamp: "Formatted at 2025-06-28 07:53:21 on dist-test-slave-wggc"
I20250628 07:53:21.173733 6434 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:21.231132 6434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:21.232662 6434 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:21.233074 6434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:21.235558 6434 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:21.239784 6434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:53:21.239996 6434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:21.240208 6434 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:53:21.240346 6434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:21.377249 6434 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:35231
I20250628 07:53:21.377368 6562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:35231 every 8 connection(s)
I20250628 07:53:21.379870 6434 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:53:21.386004 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6434
I20250628 07:53:21.386545 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:53:21.401157 6563 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:21.401645 6563 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:21.402697 6563 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:21.404974 6108 ts_manager.cc:194] Registered new tserver with Master: 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
I20250628 07:53:21.406307 6108 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:48925
I20250628 07:53:21.407258 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:21.444541 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:21.444871 3438 test_util.cc:276] Using random seed: 380159989
I20250628 07:53:21.489643 6108 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44292:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 07:53:21.536625 6365 tablet_service.cc:1468] Processing CreateTablet for tablet 3351b4de770f41679d0288327b32fcdf (DEFAULT_TABLE table=TestTable [id=b4e6650e485c410a9599cddfc4d4fbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:21.538137 6365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3351b4de770f41679d0288327b32fcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:21.559092 6583 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap starting.
I20250628 07:53:21.564659 6583 tablet_bootstrap.cc:654] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:21.566733 6583 log.cc:826] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:21.571506 6583 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: No bootstrap required, opened a new log
I20250628 07:53:21.571965 6583 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20250628 07:53:21.589339 6583 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:21.590133 6583 raft_consensus.cc:383] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:21.590463 6583 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Initialized, Role: FOLLOWER
I20250628 07:53:21.591284 6583 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:21.591820 6583 raft_consensus.cc:397] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:21.592126 6583 raft_consensus.cc:491] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:21.592487 6583 raft_consensus.cc:3058] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:21.597173 6583 raft_consensus.cc:513] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:21.597976 6583 leader_election.cc:304] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5b97082f844f69a92cb49f0407e949; no voters:
I20250628 07:53:21.599850 6583 leader_election.cc:290] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:53:21.600317 6585 raft_consensus.cc:2802] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:21.602890 6585 raft_consensus.cc:695] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 LEADER]: Becoming Leader. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Running, Role: LEADER
I20250628 07:53:21.603758 6583 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent starting tablet: real 0.032s user 0.029s sys 0.003s
I20250628 07:53:21.603974 6585 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:21.617627 6108 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5b97082f844f69a92cb49f0407e949 (127.3.91.130). New cstate: current_term: 1 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:21.862865 3438 test_util.cc:276] Using random seed: 380577854
I20250628 07:53:21.884918 6102 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44308:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 07:53:21.916081 6231 tablet_service.cc:1468] Processing CreateTablet for tablet e24b3f4c19d74938823c52ac93029c82 (DEFAULT_TABLE table=TestTable1 [id=7e6c04ace2074d9db098b86bb2c13204]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:21.917830 6231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24b3f4c19d74938823c52ac93029c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:21.939469 6604 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap starting.
I20250628 07:53:21.945549 6604 tablet_bootstrap.cc:654] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:21.947436 6604 log.cc:826] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:21.952477 6604 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: No bootstrap required, opened a new log
I20250628 07:53:21.952947 6604 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250628 07:53:21.971909 6604 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:21.972510 6604 raft_consensus.cc:383] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:21.972707 6604 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Initialized, Role: FOLLOWER
I20250628 07:53:21.973407 6604 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:21.973982 6604 raft_consensus.cc:397] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:21.974218 6604 raft_consensus.cc:491] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:21.974491 6604 raft_consensus.cc:3058] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:21.979135 6604 raft_consensus.cc:513] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:21.980078 6604 leader_election.cc:304] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff; no voters:
I20250628 07:53:21.981921 6604 leader_election.cc:290] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:53:21.982292 6606 raft_consensus.cc:2802] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:21.984992 6606 raft_consensus.cc:695] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 LEADER]: Becoming Leader. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Running, Role: LEADER
I20250628 07:53:21.986358 6604 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent starting tablet: real 0.033s user 0.026s sys 0.009s
I20250628 07:53:21.986564 6606 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:21.998206 6102 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129). New cstate: current_term: 1 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:22.200129 3438 test_util.cc:276] Using random seed: 380915237
I20250628 07:53:22.225505 6100 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44322:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250628 07:53:22.255761 6498 tablet_service.cc:1468] Processing CreateTablet for tablet cac8fad2e5f1408880c28a9299cb3d53 (DEFAULT_TABLE table=TestTable2 [id=b8a0e8a71aac4434b746478317cb787e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:53:22.257319 6498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cac8fad2e5f1408880c28a9299cb3d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:22.277762 6625 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:22.283684 6625 tablet_bootstrap.cc:654] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Neither blocks nor log segments found. Creating new log.
I20250628 07:53:22.285440 6625 log.cc:826] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:22.289876 6625 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: No bootstrap required, opened a new log
I20250628 07:53:22.290310 6625 ts_tablet_manager.cc:1397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250628 07:53:22.309111 6625 raft_consensus.cc:357] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:22.309926 6625 raft_consensus.cc:383] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:53:22.310213 6625 raft_consensus.cc:738] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: FOLLOWER
I20250628 07:53:22.311105 6625 consensus_queue.cc:260] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:22.311769 6625 raft_consensus.cc:397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:22.312106 6625 raft_consensus.cc:491] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:22.312489 6625 raft_consensus.cc:3058] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:53:22.319006 6625 raft_consensus.cc:513] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:22.319948 6625 leader_election.cc:304] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84598f3817904a8794eebb4582ab6494; no voters:
I20250628 07:53:22.322052 6625 leader_election.cc:290] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:53:22.322654 6627 raft_consensus.cc:2802] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:53:22.325834 6563 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
I20250628 07:53:22.325428 6627 raft_consensus.cc:695] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 LEADER]: Becoming Leader. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Running, Role: LEADER
I20250628 07:53:22.326829 6625 ts_tablet_manager.cc:1428] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.036s user 0.031s sys 0.005s
I20250628 07:53:22.326942 6627 consensus_queue.cc:237] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:22.339887 6100 catalog_manager.cc:5582] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84598f3817904a8794eebb4582ab6494 (127.3.91.131). New cstate: current_term: 1 leader_uuid: "84598f3817904a8794eebb4582ab6494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:22.557513 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6075
W20250628 07:53:22.636296 6430 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
W20250628 07:53:23.014278 6296 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
W20250628 07:53:23.360988 6563 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
I20250628 07:53:27.639586 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6167
I20250628 07:53:27.663784 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6300
I20250628 07:53:27.691951 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6434
I20250628 07:53:27.719785 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--webserver_interface=127.3.91.190
--webserver_port=35707
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:42237 with env {}
W20250628 07:53:28.031766 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:28.032385 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:28.032861 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:28.064239 6705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:53:28.064572 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:28.064874 6705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:53:28.065124 6705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:53:28.102679 6705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:42237
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=35707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:28.104209 6705 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:28.105949 6705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:28.119974 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:28.120008 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:28.123764 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:29.316704 6713 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:29.316856 6705 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:29.321509 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:29.324915 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:29.326368 6705 hybrid_clock.cc:648] HybridClock initialized: now 1751097209326325 us; error 61 us; skew 500 ppm
I20250628 07:53:29.327226 6705 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:29.334111 6705 webserver.cc:469] Webserver started at http://127.3.91.190:35707/ using document root <none> and password file <none>
I20250628 07:53:29.335078 6705 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:29.335302 6705 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:29.344058 6705 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.001s
I20250628 07:53:29.348783 6722 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:29.349921 6705 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250628 07:53:29.350215 6705 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:29.352193 6705 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:29.420156 6705 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:29.421876 6705 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:29.422289 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:29.493944 6705 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:42237
I20250628 07:53:29.494012 6773 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:42237 every 8 connection(s)
I20250628 07:53:29.496803 6705 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:53:29.502220 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6705
I20250628 07:53:29.503916 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:44329
--local_ip_for_outbound_sockets=127.3.91.129
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=46259
--webserver_interface=127.3.91.129
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:29.506873 6774 sys_catalog.cc:263] Verifying existing consensus state
I20250628 07:53:29.514712 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap starting.
I20250628 07:53:29.530853 6774 log.cc:826] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:29.597972 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:29.599125 6774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap complete.
I20250628 07:53:29.624434 6774 raft_consensus.cc:357] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:29.626660 6774 raft_consensus.cc:738] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Initialized, Role: FOLLOWER
I20250628 07:53:29.627600 6774 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:29.628142 6774 raft_consensus.cc:397] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:29.628404 6774 raft_consensus.cc:491] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:29.628726 6774 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:53:29.634371 6774 raft_consensus.cc:513] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:29.635030 6774 leader_election.cc:304] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a19ec6f3d445709d6936fbffe0e1fb; no voters:
I20250628 07:53:29.637257 6774 leader_election.cc:290] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 3 election: Requested vote from peers
I20250628 07:53:29.637812 6778 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Leader election won for term 3
I20250628 07:53:29.641103 6778 raft_consensus.cc:695] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 LEADER]: Becoming Leader. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Running, Role: LEADER
I20250628 07:53:29.641964 6778 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:29.642469 6774 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:53:29.653738 6779 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:29.654657 6779 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:29.657089 6780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a19ec6f3d445709d6936fbffe0e1fb. Latest consensus state: current_term: 3 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:29.657830 6780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:29.667840 6785 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:53:29.682221 6785 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=39ac042a3a7240d693a4fa697bca8b97]
I20250628 07:53:29.684612 6785 catalog_manager.cc:671] Loaded metadata for table TestTable [id=70ce0b4df1354e5f85dcbdeca770581f]
I20250628 07:53:29.686582 6785 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=b8a0e8a71aac4434b746478317cb787e]
I20250628 07:53:29.696890 6785 tablet_loader.cc:96] loaded metadata for tablet 3351b4de770f41679d0288327b32fcdf (table TestTable [id=70ce0b4df1354e5f85dcbdeca770581f])
I20250628 07:53:29.698563 6785 tablet_loader.cc:96] loaded metadata for tablet cac8fad2e5f1408880c28a9299cb3d53 (table TestTable2 [id=b8a0e8a71aac4434b746478317cb787e])
I20250628 07:53:29.700008 6785 tablet_loader.cc:96] loaded metadata for tablet e24b3f4c19d74938823c52ac93029c82 (table TestTable1 [id=39ac042a3a7240d693a4fa697bca8b97])
I20250628 07:53:29.701937 6785 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:53:29.708770 6785 catalog_manager.cc:1261] Loaded cluster ID: 4e3ba728b1464b929e513a3c12c5c940
I20250628 07:53:29.709127 6785 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:53:29.719713 6785 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:53:29.724510 6785 catalog_manager.cc:5966] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Loaded TSK: 0
I20250628 07:53:29.725769 6785 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 07:53:29.951938 6776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:29.952467 6776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:29.952977 6776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:29.984683 6776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:29.985553 6776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:53:30.022168 6776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:44329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=46259
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:30.023598 6776 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:30.025290 6776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:30.043413 6804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:30.045082 6801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:30.044015 6802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:31.244853 6803 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:31.244922 6776 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:31.250968 6776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:31.253921 6776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:31.255419 6776 hybrid_clock.cc:648] HybridClock initialized: now 1751097211255360 us; error 60 us; skew 500 ppm
I20250628 07:53:31.256543 6776 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:31.264418 6776 webserver.cc:469] Webserver started at http://127.3.91.129:46259/ using document root <none> and password file <none>
I20250628 07:53:31.265712 6776 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:31.266031 6776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:31.276836 6776 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250628 07:53:31.281785 6811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:31.283056 6776 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250628 07:53:31.283396 6776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
format_stamp: "Formatted at 2025-06-28 07:53:17 on dist-test-slave-wggc"
I20250628 07:53:31.285389 6776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:31.351794 6776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:31.353307 6776 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:31.353821 6776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:31.356451 6776 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:31.363159 6818 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:53:31.374716 6776 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:53:31.375034 6776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250628 07:53:31.375334 6776 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:53:31.380438 6776 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:53:31.380661 6776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250628 07:53:31.381155 6818 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap starting.
I20250628 07:53:31.458729 6818 log.cc:826] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:31.582137 6818 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:31.583133 6818 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap complete.
I20250628 07:53:31.584861 6818 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent bootstrapping tablet: real 0.204s user 0.157s sys 0.036s
I20250628 07:53:31.607164 6818 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:31.610435 6818 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Initialized, Role: FOLLOWER
I20250628 07:53:31.611627 6818 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:31.612287 6818 raft_consensus.cc:397] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:31.612593 6818 raft_consensus.cc:491] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:31.612975 6818 raft_consensus.cc:3058] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:53:31.621556 6818 raft_consensus.cc:513] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:31.622766 6818 leader_election.cc:304] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff; no voters:
I20250628 07:53:31.626356 6818 leader_election.cc:290] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 07:53:31.626785 6925 raft_consensus.cc:2802] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:53:31.633105 6776 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:44329
I20250628 07:53:31.633277 6926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:44329 every 8 connection(s)
I20250628 07:53:31.637916 6776 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:53:31.639703 6925 raft_consensus.cc:695] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEADER]: Becoming Leader. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Running, Role: LEADER
I20250628 07:53:31.641794 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6776
I20250628 07:53:31.643157 6818 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent starting tablet: real 0.058s user 0.048s sys 0.008s
I20250628 07:53:31.644299 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:33649
--local_ip_for_outbound_sockets=127.3.91.130
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=37845
--webserver_interface=127.3.91.130
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:31.649952 6925 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } }
I20250628 07:53:31.705257 6929 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:31.705801 6929 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:31.707060 6929 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:31.712157 6739 ts_manager.cc:194] Registered new tserver with Master: 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329)
I20250628 07:53:31.716117 6739 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: term changed from 0 to 2, leader changed from <none> to 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129), VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) added. New cstate: current_term: 2 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } }
W20250628 07:53:31.762522 6739 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e24b3f4c19d74938823c52ac93029c82 with cas_config_opid_index -1: no extra replica candidate found for tablet e24b3f4c19d74938823c52ac93029c82 (table TestTable1 [id=39ac042a3a7240d693a4fa697bca8b97]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250628 07:53:31.764762 6739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:54193
I20250628 07:53:31.769037 6929 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
W20250628 07:53:32.012378 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:32.012921 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:32.013813 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:32.048549 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:32.049448 6932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:53:32.085292 6932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:33649
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=37845
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:32.086686 6932 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:32.089057 6932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:32.114464 6948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:32.114464 6946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:32.115448 6945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:33.277765 6947 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:33.277882 6932 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:33.281999 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:33.284741 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:33.286190 6932 hybrid_clock.cc:648] HybridClock initialized: now 1751097213286154 us; error 56 us; skew 500 ppm
I20250628 07:53:33.286994 6932 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:33.293759 6932 webserver.cc:469] Webserver started at http://127.3.91.130:37845/ using document root <none> and password file <none>
I20250628 07:53:33.294675 6932 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:33.294896 6932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:33.302603 6932 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250628 07:53:33.307632 6955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:33.308663 6932 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 07:53:33.308993 6932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce5b97082f844f69a92cb49f0407e949"
format_stamp: "Formatted at 2025-06-28 07:53:19 on dist-test-slave-wggc"
I20250628 07:53:33.311020 6932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:33.366549 6932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:33.368052 6932 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:33.368515 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:33.371182 6932 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:33.377043 6962 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:53:33.384754 6932 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:53:33.385016 6932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250628 07:53:33.385319 6932 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:53:33.390311 6932 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:53:33.390515 6932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250628 07:53:33.390913 6962 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap starting.
I20250628 07:53:33.462006 6962 log.cc:826] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:33.559752 6932 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:33649
I20250628 07:53:33.559804 7069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:33649 every 8 connection(s)
I20250628 07:53:33.563441 6932 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:53:33.565948 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 6932
I20250628 07:53:33.567879 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:35231
--local_ip_for_outbound_sockets=127.3.91.131
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=46791
--webserver_interface=127.3.91.131
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:33.600836 7070 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:33.601269 7070 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:33.602264 7070 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:33.605204 6739 ts_manager.cc:194] Registered new tserver with Master: ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649)
I20250628 07:53:33.607672 6739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:44827
I20250628 07:53:33.610723 6962 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:33.611660 6962 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap complete.
I20250628 07:53:33.613391 6962 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent bootstrapping tablet: real 0.223s user 0.168s sys 0.048s
I20250628 07:53:33.630115 6962 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:33.633020 6962 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Initialized, Role: FOLLOWER
I20250628 07:53:33.633879 6962 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:33.634521 6962 raft_consensus.cc:397] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:33.634866 6962 raft_consensus.cc:491] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:33.635257 6962 raft_consensus.cc:3058] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:53:33.643142 6962 raft_consensus.cc:513] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:33.644037 6962 leader_election.cc:304] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5b97082f844f69a92cb49f0407e949; no voters:
I20250628 07:53:33.646277 6962 leader_election.cc:290] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 07:53:33.646735 7075 raft_consensus.cc:2802] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:53:33.650002 7070 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
I20250628 07:53:33.650342 6962 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent starting tablet: real 0.037s user 0.019s sys 0.016s
I20250628 07:53:33.651700 7075 raft_consensus.cc:695] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 LEADER]: Becoming Leader. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Running, Role: LEADER
I20250628 07:53:33.652464 7075 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:33.664223 6739 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: term changed from 0 to 2, leader changed from <none> to ce5b97082f844f69a92cb49f0407e949 (127.3.91.130), VOTER ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) added. New cstate: current_term: 2 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:33.694161 7025 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: NON_VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: true } }
I20250628 07:53:33.697463 7077 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: NON_VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: true } } }
I20250628 07:53:33.707474 6723 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3351b4de770f41679d0288327b32fcdf with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 07:53:33.710618 6739 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: config changed from index -1 to 9, NON_VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) added. New cstate: current_term: 2 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: NON_VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 07:53:33.714520 6957 consensus_peers.cc:489] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 -> Peer 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329): Couldn't send request to peer 68ea013ff1aa47f996b38a6cc264a7ff. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3351b4de770f41679d0288327b32fcdf. This is attempt 1: this message will repeat every 5th retry.
W20250628 07:53:33.913273 7074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:33.913827 7074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:33.914304 7074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:33.947068 7074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:33.947886 7074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:53:33.982951 7074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:35231
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=46791
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:33.984467 7074 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:33.986167 7074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:34.003934 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:34.076570 6881 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: NON_VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: true } }
I20250628 07:53:34.084568 7099 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: NON_VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: true } } }
I20250628 07:53:34.093825 6724 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e24b3f4c19d74938823c52ac93029c82 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250628 07:53:34.098673 6739 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index -1 to 9, NON_VOTER ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) added. New cstate: current_term: 2 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: NON_VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250628 07:53:34.106380 6812 consensus_peers.cc:489] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff -> Peer ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649): Couldn't send request to peer ce5b97082f844f69a92cb49f0407e949. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e24b3f4c19d74938823c52ac93029c82. This is attempt 1: this message will repeat every 5th retry.
I20250628 07:53:34.210804 7102 ts_tablet_manager.cc:927] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Initiating tablet copy from peer ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649)
I20250628 07:53:34.213286 7102 tablet_copy_client.cc:323] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: tablet copy: Beginning tablet copy session from remote peer at address 127.3.91.130:33649
I20250628 07:53:34.215263 7045 tablet_copy_service.cc:140] P ce5b97082f844f69a92cb49f0407e949: Received BeginTabletCopySession request for tablet 3351b4de770f41679d0288327b32fcdf from peer 68ea013ff1aa47f996b38a6cc264a7ff ({username='slave'} at 127.3.91.129:40763)
I20250628 07:53:34.215776 7045 tablet_copy_service.cc:161] P ce5b97082f844f69a92cb49f0407e949: Beginning new tablet copy session on tablet 3351b4de770f41679d0288327b32fcdf from peer 68ea013ff1aa47f996b38a6cc264a7ff at {username='slave'} at 127.3.91.129:40763: session id = 68ea013ff1aa47f996b38a6cc264a7ff-3351b4de770f41679d0288327b32fcdf
I20250628 07:53:34.221148 7045 tablet_copy_source_session.cc:215] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 07:53:34.225286 7102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3351b4de770f41679d0288327b32fcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:34.235809 7102 tablet_copy_client.cc:806] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: tablet copy: Starting download of 0 data blocks...
I20250628 07:53:34.236373 7102 tablet_copy_client.cc:670] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: tablet copy: Starting download of 1 WAL segments...
I20250628 07:53:34.240978 7102 tablet_copy_client.cc:538] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 07:53:34.247228 7102 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap starting.
I20250628 07:53:34.330053 7102 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:34.330714 7102 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap complete.
I20250628 07:53:34.331146 7102 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent bootstrapping tablet: real 0.084s user 0.070s sys 0.013s
I20250628 07:53:34.332974 7102 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: NON_VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: true } }
I20250628 07:53:34.333417 7102 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Initialized, Role: LEARNER
I20250628 07:53:34.333875 7102 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: NON_VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: true } }
I20250628 07:53:34.335484 7102 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250628 07:53:34.337301 7045 tablet_copy_service.cc:342] P ce5b97082f844f69a92cb49f0407e949: Request end of tablet copy session 68ea013ff1aa47f996b38a6cc264a7ff-3351b4de770f41679d0288327b32fcdf received from {username='slave'} at 127.3.91.129:40763
I20250628 07:53:34.337749 7045 tablet_copy_service.cc:434] P ce5b97082f844f69a92cb49f0407e949: ending tablet copy session 68ea013ff1aa47f996b38a6cc264a7ff-3351b4de770f41679d0288327b32fcdf on tablet 3351b4de770f41679d0288327b32fcdf with peer 68ea013ff1aa47f996b38a6cc264a7ff
I20250628 07:53:34.602281 7105 ts_tablet_manager.cc:927] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Initiating tablet copy from peer 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329)
I20250628 07:53:34.605202 7105 tablet_copy_client.cc:323] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: tablet copy: Beginning tablet copy session from remote peer at address 127.3.91.129:44329
I20250628 07:53:34.607699 6901 tablet_copy_service.cc:140] P 68ea013ff1aa47f996b38a6cc264a7ff: Received BeginTabletCopySession request for tablet e24b3f4c19d74938823c52ac93029c82 from peer ce5b97082f844f69a92cb49f0407e949 ({username='slave'} at 127.3.91.130:43977)
I20250628 07:53:34.608615 6901 tablet_copy_service.cc:161] P 68ea013ff1aa47f996b38a6cc264a7ff: Beginning new tablet copy session on tablet e24b3f4c19d74938823c52ac93029c82 from peer ce5b97082f844f69a92cb49f0407e949 at {username='slave'} at 127.3.91.130:43977: session id = ce5b97082f844f69a92cb49f0407e949-e24b3f4c19d74938823c52ac93029c82
I20250628 07:53:34.620005 6901 tablet_copy_source_session.cc:215] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 07:53:34.625260 7105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24b3f4c19d74938823c52ac93029c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:34.641036 7105 tablet_copy_client.cc:806] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: tablet copy: Starting download of 0 data blocks...
I20250628 07:53:34.641722 7105 tablet_copy_client.cc:670] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: tablet copy: Starting download of 1 WAL segments...
I20250628 07:53:34.647279 7105 tablet_copy_client.cc:538] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 07:53:34.655781 7105 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap starting.
I20250628 07:53:34.745219 7105 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:34.746155 7105 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap complete.
I20250628 07:53:34.746737 7105 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Time spent bootstrapping tablet: real 0.091s user 0.075s sys 0.017s
I20250628 07:53:34.748395 7105 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: NON_VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: true } }
I20250628 07:53:34.748827 7105 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Initialized, Role: LEARNER
I20250628 07:53:34.749228 7105 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: NON_VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: true } }
I20250628 07:53:34.751997 7105 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250628 07:53:34.753441 6901 tablet_copy_service.cc:342] P 68ea013ff1aa47f996b38a6cc264a7ff: Request end of tablet copy session ce5b97082f844f69a92cb49f0407e949-e24b3f4c19d74938823c52ac93029c82 received from {username='slave'} at 127.3.91.130:43977
I20250628 07:53:34.753926 6901 tablet_copy_service.cc:434] P 68ea013ff1aa47f996b38a6cc264a7ff: ending tablet copy session ce5b97082f844f69a92cb49f0407e949-e24b3f4c19d74938823c52ac93029c82 on tablet e24b3f4c19d74938823c52ac93029c82 with peer ce5b97082f844f69a92cb49f0407e949
I20250628 07:53:34.832002 6881 raft_consensus.cc:1215] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250628 07:53:35.167596 7025 raft_consensus.cc:1215] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250628 07:53:34.004047 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:34.006143 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:35.209190 7094 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:53:35.209213 7074 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:53:35.213740 7074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:35.215941 7074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:35.217319 7074 hybrid_clock.cc:648] HybridClock initialized: now 1751097215217278 us; error 49 us; skew 500 ppm
I20250628 07:53:35.218168 7074 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:35.227545 7074 webserver.cc:469] Webserver started at http://127.3.91.131:46791/ using document root <none> and password file <none>
I20250628 07:53:35.228523 7074 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:35.228781 7074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:35.236795 7074 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250628 07:53:35.241585 7117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:35.242764 7074 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250628 07:53:35.243090 7074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "84598f3817904a8794eebb4582ab6494"
format_stamp: "Formatted at 2025-06-28 07:53:21 on dist-test-slave-wggc"
I20250628 07:53:35.245201 7074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:35.304232 7074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:35.305840 7074 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:35.306335 7074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:35.308910 7074 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:35.314893 7124 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:53:35.322858 7074 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:53:35.323117 7074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250628 07:53:35.323414 7074 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:53:35.328250 7074 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:53:35.328482 7074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.001s
I20250628 07:53:35.328950 7124 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:35.389371 7124 log.cc:826] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:35.436467 7108 raft_consensus.cc:1062] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: attempting to promote NON_VOTER 68ea013ff1aa47f996b38a6cc264a7ff to VOTER
I20250628 07:53:35.439072 7108 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } }
I20250628 07:53:35.445730 6881 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEARNER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250628 07:53:35.447515 7108 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250628 07:53:35.458191 7184 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:35.459852 6881 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:35.472568 6739 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index 9 to 10, 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:35.507203 7124 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:35.507988 7124 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:35.509651 7124 ts_tablet_manager.cc:1397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.181s user 0.155s sys 0.018s
I20250628 07:53:35.535665 7124 raft_consensus.cc:357] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:35.539219 7124 raft_consensus.cc:738] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: FOLLOWER
I20250628 07:53:35.540089 7124 consensus_queue.cc:260] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:35.540849 7124 raft_consensus.cc:397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:35.541180 7124 raft_consensus.cc:491] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:35.541635 7124 raft_consensus.cc:3058] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:53:35.546830 7074 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:35231
I20250628 07:53:35.546974 7238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:35231 every 8 connection(s)
I20250628 07:53:35.550272 7074 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:53:35.550267 7124 raft_consensus.cc:513] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:35.551110 7124 leader_election.cc:304] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84598f3817904a8794eebb4582ab6494; no voters:
I20250628 07:53:35.553669 7124 leader_election.cc:290] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 07:53:35.554124 7240 raft_consensus.cc:2802] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:53:35.556347 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7074
I20250628 07:53:35.563272 7240 raft_consensus.cc:695] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 LEADER]: Becoming Leader. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Running, Role: LEADER
I20250628 07:53:35.564520 7240 consensus_queue.cc:237] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:35.603957 7124 ts_tablet_manager.cc:1428] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.094s user 0.037s sys 0.039s
I20250628 07:53:35.619227 7112 raft_consensus.cc:1062] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: attempting to promote NON_VOTER ce5b97082f844f69a92cb49f0407e949 to VOTER
I20250628 07:53:35.621091 7112 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } }
I20250628 07:53:35.621572 7239 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:35.622088 7239 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:35.623412 7239 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:35.628129 6739 ts_manager.cc:194] Registered new tserver with Master: 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
I20250628 07:53:35.629643 6739 catalog_manager.cc:5582] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "84598f3817904a8794eebb4582ab6494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:35.635025 7025 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 LEARNER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250628 07:53:35.636863 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:35.636722 7099 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250628 07:53:35.639184 6739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:45289
I20250628 07:53:35.642442 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:35.643353 7239 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
W20250628 07:53:35.648051 3438 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 3351b4de770f41679d0288327b32fcdf: tablet_id: "3351b4de770f41679d0288327b32fcdf" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250628 07:53:35.651191 7099 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } }
I20250628 07:53:35.652632 7025 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } }
I20250628 07:53:35.662012 6738 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 reported cstate change: config changed from index 9 to 10, ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } }
I20250628 07:53:35.674074 6881 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:35.681234 7025 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250628 07:53:35.682808 7111 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250628 07:53:35.683840 7024 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:35.690220 6881 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250628 07:53:35.692023 7184 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250628 07:53:35.697583 7111 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } } }
W20250628 07:53:35.698748 6815 consensus_peers.cc:489] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff -> Peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Couldn't send request to peer 84598f3817904a8794eebb4582ab6494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e24b3f4c19d74938823c52ac93029c82. This is attempt 1: this message will repeat every 5th retry.
I20250628 07:53:35.699801 7108 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } } }
I20250628 07:53:35.702405 6881 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } } }
I20250628 07:53:35.706872 6724 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e24b3f4c19d74938823c52ac93029c82 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250628 07:53:35.700261 7025 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } } }
W20250628 07:53:35.710917 6959 consensus_peers.cc:489] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 -> Peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Couldn't send request to peer 84598f3817904a8794eebb4582ab6494. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3351b4de770f41679d0288327b32fcdf. This is attempt 1: this message will repeat every 5th retry.
I20250628 07:53:35.711400 6723 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3351b4de770f41679d0288327b32fcdf with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250628 07:53:35.713667 6739 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New cstate: current_term: 2 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250628 07:53:35.716455 6738 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: config changed from index 10 to 11, NON_VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added. New cstate: current_term: 2 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250628 07:53:35.912030 6725 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3351b4de770f41679d0288327b32fcdf with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250628 07:53:36.129545 7255 ts_tablet_manager.cc:927] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Initiating tablet copy from peer 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329)
I20250628 07:53:36.131489 7255 tablet_copy_client.cc:323] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: tablet copy: Beginning tablet copy session from remote peer at address 127.3.91.129:44329
I20250628 07:53:36.141081 6901 tablet_copy_service.cc:140] P 68ea013ff1aa47f996b38a6cc264a7ff: Received BeginTabletCopySession request for tablet e24b3f4c19d74938823c52ac93029c82 from peer 84598f3817904a8794eebb4582ab6494 ({username='slave'} at 127.3.91.131:46635)
I20250628 07:53:36.141471 6901 tablet_copy_service.cc:161] P 68ea013ff1aa47f996b38a6cc264a7ff: Beginning new tablet copy session on tablet e24b3f4c19d74938823c52ac93029c82 from peer 84598f3817904a8794eebb4582ab6494 at {username='slave'} at 127.3.91.131:46635: session id = 84598f3817904a8794eebb4582ab6494-e24b3f4c19d74938823c52ac93029c82
I20250628 07:53:36.145990 6901 tablet_copy_source_session.cc:215] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 07:53:36.148842 7255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e24b3f4c19d74938823c52ac93029c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:36.161494 7255 tablet_copy_client.cc:806] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: tablet copy: Starting download of 0 data blocks...
I20250628 07:53:36.162034 7255 tablet_copy_client.cc:670] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: tablet copy: Starting download of 1 WAL segments...
I20250628 07:53:36.165700 7255 tablet_copy_client.cc:538] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 07:53:36.171334 7255 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:36.231362 7261 ts_tablet_manager.cc:927] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Initiating tablet copy from peer ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649)
I20250628 07:53:36.233788 7261 tablet_copy_client.cc:323] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: tablet copy: Beginning tablet copy session from remote peer at address 127.3.91.130:33649
I20250628 07:53:36.243415 7045 tablet_copy_service.cc:140] P ce5b97082f844f69a92cb49f0407e949: Received BeginTabletCopySession request for tablet 3351b4de770f41679d0288327b32fcdf from peer 84598f3817904a8794eebb4582ab6494 ({username='slave'} at 127.3.91.131:48981)
I20250628 07:53:36.243825 7045 tablet_copy_service.cc:161] P ce5b97082f844f69a92cb49f0407e949: Beginning new tablet copy session on tablet 3351b4de770f41679d0288327b32fcdf from peer 84598f3817904a8794eebb4582ab6494 at {username='slave'} at 127.3.91.131:48981: session id = 84598f3817904a8794eebb4582ab6494-3351b4de770f41679d0288327b32fcdf
I20250628 07:53:36.248229 7045 tablet_copy_source_session.cc:215] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Tablet Copy: opened 0 blocks and 1 log segments
I20250628 07:53:36.250555 7261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3351b4de770f41679d0288327b32fcdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:53:36.256943 7255 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:36.257550 7255 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:36.258078 7255 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.087s user 0.079s sys 0.008s
I20250628 07:53:36.259651 7255 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:36.260124 7255 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: LEARNER
I20250628 07:53:36.260236 7261 tablet_copy_client.cc:806] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: tablet copy: Starting download of 0 data blocks...
I20250628 07:53:36.260766 7261 tablet_copy_client.cc:670] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: tablet copy: Starting download of 1 WAL segments...
I20250628 07:53:36.260706 7255 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:36.264052 7255 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250628 07:53:36.264333 7261 tablet_copy_client.cc:538] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250628 07:53:36.265539 6901 tablet_copy_service.cc:342] P 68ea013ff1aa47f996b38a6cc264a7ff: Request end of tablet copy session 84598f3817904a8794eebb4582ab6494-e24b3f4c19d74938823c52ac93029c82 received from {username='slave'} at 127.3.91.131:46635
I20250628 07:53:36.265969 6901 tablet_copy_service.cc:434] P 68ea013ff1aa47f996b38a6cc264a7ff: ending tablet copy session 84598f3817904a8794eebb4582ab6494-e24b3f4c19d74938823c52ac93029c82 on tablet e24b3f4c19d74938823c52ac93029c82 with peer 84598f3817904a8794eebb4582ab6494
I20250628 07:53:36.271732 7261 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:36.334022 6723 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e24b3f4c19d74938823c52ac93029c82 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250628 07:53:36.354981 7261 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:36.355607 7261 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:36.356042 7261 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.084s user 0.060s sys 0.024s
I20250628 07:53:36.357580 7261 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:36.358129 7261 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: LEARNER
I20250628 07:53:36.358531 7261 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: NON_VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: true } }
I20250628 07:53:36.360136 7261 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250628 07:53:36.361724 7045 tablet_copy_service.cc:342] P ce5b97082f844f69a92cb49f0407e949: Request end of tablet copy session 84598f3817904a8794eebb4582ab6494-3351b4de770f41679d0288327b32fcdf received from {username='slave'} at 127.3.91.131:48981
I20250628 07:53:36.362023 7045 tablet_copy_service.cc:434] P ce5b97082f844f69a92cb49f0407e949: ending tablet copy session 84598f3817904a8794eebb4582ab6494-3351b4de770f41679d0288327b32fcdf on tablet 3351b4de770f41679d0288327b32fcdf with peer 84598f3817904a8794eebb4582ab6494
I20250628 07:53:36.604326 7188 raft_consensus.cc:1215] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250628 07:53:36.611174 7188 raft_consensus.cc:1215] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250628 07:53:36.652936 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 68ea013ff1aa47f996b38a6cc264a7ff to finish bootstrapping
I20250628 07:53:36.666301 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce5b97082f844f69a92cb49f0407e949 to finish bootstrapping
I20250628 07:53:36.677553 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 84598f3817904a8794eebb4582ab6494 to finish bootstrapping
I20250628 07:53:36.935974 6861 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:53:36.944802 7004 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:53:36.951413 7167 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:53:37.027535 7269 raft_consensus.cc:1062] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: attempting to promote NON_VOTER 84598f3817904a8794eebb4582ab6494 to VOTER
I20250628 07:53:37.029522 7269 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:37.046752 7188 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 07:53:37.048516 7260 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250628 07:53:37.056191 6881 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 07:53:37.058090 7260 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250628 07:53:37.060951 7268 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.067164 6880 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.073742 6738 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250628 07:53:37.077271 7187 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.118309 7267 raft_consensus.cc:1062] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: attempting to promote NON_VOTER 84598f3817904a8794eebb4582ab6494 to VOTER
I20250628 07:53:37.120477 7267 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:37.130314 7187 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 LEARNER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 07:53:37.136894 7303 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250628 07:53:37.137379 7025 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250628 07:53:37.140075 7267 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250628 07:53:37.163303 7308 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.170543 7025 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.189455 7187 raft_consensus.cc:2953] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:37.192873 6737 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index 11 to 12, 84598f3817904a8794eebb4582ab6494 (127.3.91.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b7a19ec6f3d445709d6936fbffe0e1fb | 127.3.91.190:42237 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:34015 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
68ea013ff1aa47f996b38a6cc264a7ff | 127.3.91.129:44329 | HEALTHY | <none> | 1 | 0
84598f3817904a8794eebb4582ab6494 | 127.3.91.131:35231 | HEALTHY | <none> | 1 | 0
ce5b97082f844f69a92cb49f0407e949 | 127.3.91.130:33649 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.91.129 | experimental | 127.3.91.129:44329
local_ip_for_outbound_sockets | 127.3.91.130 | experimental | 127.3.91.130:33649
local_ip_for_outbound_sockets | 127.3.91.131 | experimental | 127.3.91.131:35231
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.3.91.129:44329
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.3.91.130:33649
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.3.91.131:35231
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:34015 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 07:53:37.252069 3438 log_verifier.cc:126] Checking tablet 3351b4de770f41679d0288327b32fcdf
I20250628 07:53:37.346985 3438 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 3351b4de770f41679d0288327b32fcdf
I20250628 07:53:37.347295 3438 log_verifier.cc:126] Checking tablet cac8fad2e5f1408880c28a9299cb3d53
I20250628 07:53:37.374636 3438 log_verifier.cc:177] Verified matching terms for 8 ops in tablet cac8fad2e5f1408880c28a9299cb3d53
I20250628 07:53:37.374850 3438 log_verifier.cc:126] Checking tablet e24b3f4c19d74938823c52ac93029c82
I20250628 07:53:37.458742 3438 log_verifier.cc:177] Verified matching terms for 12 ops in tablet e24b3f4c19d74938823c52ac93029c82
I20250628 07:53:37.459115 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6705
I20250628 07:53:37.488667 3438 minidump.cc:252] Setting minidump size limit to 20M
I20250628 07:53:37.489904 3438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:37.490883 3438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:37.501161 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:37.502666 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:37.501221 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:37.664331 3438 server_base.cc:1048] running on GCE node
I20250628 07:53:37.665868 3438 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250628 07:53:37.666155 3438 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250628 07:53:37.666356 3438 hybrid_clock.cc:648] HybridClock initialized: now 1751097217666337 us; error 0 us; skew 500 ppm
I20250628 07:53:37.667215 3438 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:37.670765 3438 webserver.cc:469] Webserver started at http://0.0.0.0:35501/ using document root <none> and password file <none>
I20250628 07:53:37.671794 3438 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:37.672041 3438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:37.678674 3438 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250628 07:53:37.683249 7320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:37.684271 3438 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250628 07:53:37.684641 3438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:37.686995 3438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:37.702795 3438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:37.704620 3438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:37.705150 3438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:37.716631 3438 sys_catalog.cc:263] Verifying existing consensus state
W20250628 07:53:37.720134 3438 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.3.91.190:42237 exists but no master address supplied!
I20250628 07:53:37.722271 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap starting.
I20250628 07:53:37.765101 3438 log.cc:826] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:37.827441 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:37.828218 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap complete.
I20250628 07:53:37.841534 3438 raft_consensus.cc:357] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:37.842141 3438 raft_consensus.cc:738] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Initialized, Role: FOLLOWER
I20250628 07:53:37.842994 3438 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:37.843465 3438 raft_consensus.cc:397] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:37.843714 3438 raft_consensus.cc:491] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:37.844029 3438 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 3 FOLLOWER]: Advancing to term 4
I20250628 07:53:37.849421 3438 raft_consensus.cc:513] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:37.850160 3438 leader_election.cc:304] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a19ec6f3d445709d6936fbffe0e1fb; no voters:
I20250628 07:53:37.851258 3438 leader_election.cc:290] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 4 election: Requested vote from peers
I20250628 07:53:37.851538 7328 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 4 FOLLOWER]: Leader election won for term 4
I20250628 07:53:37.852916 7328 raft_consensus.cc:695] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 4 LEADER]: Becoming Leader. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Running, Role: LEADER
I20250628 07:53:37.853864 7328 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:37.860968 7330 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:37.861747 7330 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:37.861740 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a19ec6f3d445709d6936fbffe0e1fb. Latest consensus state: current_term: 4 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:37.862243 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:37.889745 3438 tablet_replica.cc:331] stopping tablet replica
I20250628 07:53:37.890326 3438 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 4 LEADER]: Raft consensus shutting down.
I20250628 07:53:37.890749 3438 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 4 FOLLOWER]: Raft consensus is shut down!
I20250628 07:53:37.893180 3438 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 07:53:37.893759 3438 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 07:53:37.918393 3438 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250628 07:53:38.231678 7070 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
W20250628 07:53:38.232226 7239 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
W20250628 07:53:38.235332 6929 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:42237 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:42237: connect: Connection refused (error 111)
I20250628 07:53:43.224059 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6776
I20250628 07:53:43.251389 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 6932
I20250628 07:53:43.281443 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7074
I20250628 07:53:43.311656 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--webserver_interface=127.3.91.190
--webserver_port=35707
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:42237 with env {}
W20250628 07:53:43.621825 7407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:43.622445 7407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:43.622974 7407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:43.653558 7407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:53:43.653926 7407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:43.654259 7407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:53:43.654667 7407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:53:43.692996 7407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:42237
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:42237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=35707
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:43.694299 7407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:43.695880 7407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:43.711863 7414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:43.713138 7416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:43.713311 7407 server_base.cc:1048] running on GCE node
W20250628 07:53:43.712395 7413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:44.915247 7407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:44.917939 7407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:44.919344 7407 hybrid_clock.cc:648] HybridClock initialized: now 1751097224919287 us; error 73 us; skew 500 ppm
I20250628 07:53:44.920325 7407 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:44.927770 7407 webserver.cc:469] Webserver started at http://127.3.91.190:35707/ using document root <none> and password file <none>
I20250628 07:53:44.928776 7407 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:44.928992 7407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:44.937371 7407 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250628 07:53:44.942353 7423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:44.943564 7407 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 07:53:44.943945 7407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b7a19ec6f3d445709d6936fbffe0e1fb"
format_stamp: "Formatted at 2025-06-28 07:53:15 on dist-test-slave-wggc"
I20250628 07:53:44.945973 7407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:45.023346 7407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:45.024827 7407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:45.025290 7407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:45.098210 7407 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:42237
I20250628 07:53:45.098287 7474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:42237 every 8 connection(s)
I20250628 07:53:45.101243 7407 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:53:45.105172 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7407
I20250628 07:53:45.107250 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:44329
--local_ip_for_outbound_sockets=127.3.91.129
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=46259
--webserver_interface=127.3.91.129
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:45.116925 7475 sys_catalog.cc:263] Verifying existing consensus state
I20250628 07:53:45.123325 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap starting.
I20250628 07:53:45.133824 7475 log.cc:826] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:45.215562 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:45.216398 7475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Bootstrap complete.
I20250628 07:53:45.235914 7475 raft_consensus.cc:357] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:45.238297 7475 raft_consensus.cc:738] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Initialized, Role: FOLLOWER
I20250628 07:53:45.239117 7475 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:45.239573 7475 raft_consensus.cc:397] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:45.239795 7475 raft_consensus.cc:491] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:45.240051 7475 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 5 FOLLOWER]: Advancing to term 6
I20250628 07:53:45.245285 7475 raft_consensus.cc:513] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:45.245985 7475 leader_election.cc:304] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a19ec6f3d445709d6936fbffe0e1fb; no voters:
I20250628 07:53:45.248149 7475 leader_election.cc:290] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [CANDIDATE]: Term 6 election: Requested vote from peers
I20250628 07:53:45.248648 7479 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 6 FOLLOWER]: Leader election won for term 6
I20250628 07:53:45.251983 7479 raft_consensus.cc:695] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [term 6 LEADER]: Becoming Leader. State: Replica: b7a19ec6f3d445709d6936fbffe0e1fb, State: Running, Role: LEADER
I20250628 07:53:45.252933 7479 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } }
I20250628 07:53:45.253360 7475 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:53:45.263949 7480 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:45.265872 7480 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:45.265197 7481 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a19ec6f3d445709d6936fbffe0e1fb. Latest consensus state: current_term: 6 leader_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a19ec6f3d445709d6936fbffe0e1fb" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 42237 } } }
I20250628 07:53:45.267680 7481 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb [sys.catalog]: This master's current role is: LEADER
I20250628 07:53:45.277585 7486 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:53:45.298949 7486 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=39ac042a3a7240d693a4fa697bca8b97]
I20250628 07:53:45.300843 7486 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=b8a0e8a71aac4434b746478317cb787e]
I20250628 07:53:45.302605 7486 catalog_manager.cc:671] Loaded metadata for table TestTable [id=fb4b1a3f48224a4f8b103d900fcfaffe]
I20250628 07:53:45.312353 7486 tablet_loader.cc:96] loaded metadata for tablet 3351b4de770f41679d0288327b32fcdf (table TestTable [id=fb4b1a3f48224a4f8b103d900fcfaffe])
I20250628 07:53:45.313928 7486 tablet_loader.cc:96] loaded metadata for tablet cac8fad2e5f1408880c28a9299cb3d53 (table TestTable2 [id=b8a0e8a71aac4434b746478317cb787e])
I20250628 07:53:45.315600 7486 tablet_loader.cc:96] loaded metadata for tablet e24b3f4c19d74938823c52ac93029c82 (table TestTable1 [id=39ac042a3a7240d693a4fa697bca8b97])
I20250628 07:53:45.317188 7486 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:53:45.324218 7486 catalog_manager.cc:1261] Loaded cluster ID: 4e3ba728b1464b929e513a3c12c5c940
I20250628 07:53:45.324803 7486 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:53:45.334136 7486 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:53:45.341742 7486 catalog_manager.cc:5966] T 00000000000000000000000000000000 P b7a19ec6f3d445709d6936fbffe0e1fb: Loaded TSK: 0
I20250628 07:53:45.344035 7486 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 07:53:45.484809 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:45.485378 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:45.485965 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:45.519742 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:45.520614 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:53:45.556843 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:44329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=46259
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:45.558354 7477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:45.560086 7477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:45.578850 7503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:45.582661 7505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:45.583051 7477 server_base.cc:1048] running on GCE node
W20250628 07:53:45.582818 7502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:46.802479 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:46.805513 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:46.807060 7477 hybrid_clock.cc:648] HybridClock initialized: now 1751097226806989 us; error 89 us; skew 500 ppm
I20250628 07:53:46.808068 7477 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:46.816927 7477 webserver.cc:469] Webserver started at http://127.3.91.129:46259/ using document root <none> and password file <none>
I20250628 07:53:46.818048 7477 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:46.818289 7477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:46.827416 7477 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.007s
I20250628 07:53:46.833374 7512 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:46.834721 7477 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250628 07:53:46.835098 7477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
format_stamp: "Formatted at 2025-06-28 07:53:17 on dist-test-slave-wggc"
I20250628 07:53:46.837178 7477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:46.895037 7477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:46.896703 7477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:46.897188 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:46.899981 7477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:46.906936 7519 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250628 07:53:46.926117 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250628 07:53:46.926489 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.000s sys 0.002s
I20250628 07:53:46.926853 7477 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250628 07:53:46.935662 7519 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap starting.
I20250628 07:53:46.941004 7477 ts_tablet_manager.cc:610] Registered 2 tablets
I20250628 07:53:46.941320 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.015s user 0.011s sys 0.001s
I20250628 07:53:47.010995 7519 log.cc:826] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:47.151206 7519 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:47.152406 7519 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap complete.
I20250628 07:53:47.154368 7519 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent bootstrapping tablet: real 0.219s user 0.168s sys 0.044s
I20250628 07:53:47.177225 7519 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:47.180809 7519 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Initialized, Role: FOLLOWER
I20250628 07:53:47.183148 7519 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:47.190186 7519 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent starting tablet: real 0.036s user 0.029s sys 0.005s
I20250628 07:53:47.191123 7519 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap starting.
I20250628 07:53:47.203612 7477 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:44329
I20250628 07:53:47.203784 7627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:44329 every 8 connection(s)
I20250628 07:53:47.206516 7477 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:53:47.211846 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7477
I20250628 07:53:47.214289 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:33649
--local_ip_for_outbound_sockets=127.3.91.130
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=37845
--webserver_interface=127.3.91.130
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:47.268522 7628 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:47.269035 7628 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:47.270360 7628 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:47.276067 7440 ts_manager.cc:194] Registered new tserver with Master: 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329)
I20250628 07:53:47.281524 7440 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) added, VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) added, VOTER ce5b97082f844f69a92cb49f0407e949 (127.3.91.130) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } } }
I20250628 07:53:47.354394 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:46777
I20250628 07:53:47.373539 7519 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:47.374606 7519 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Bootstrap complete.
I20250628 07:53:47.376330 7519 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent bootstrapping tablet: real 0.185s user 0.143s sys 0.032s
I20250628 07:53:47.378908 7519 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:47.379706 7519 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Initialized, Role: FOLLOWER
I20250628 07:53:47.380431 7519 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:47.381819 7628 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
I20250628 07:53:47.382733 7519 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20250628 07:53:47.584352 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:47.584805 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:47.585253 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:47.616089 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:47.616902 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:53:47.651160 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:33649
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=37845
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:47.652441 7630 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:47.654131 7630 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:47.671388 7644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:48.523897 7649 raft_consensus.cc:491] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:53:48.524421 7649 raft_consensus.cc:513] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:48.526738 7649 leader_election.cc:290] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
W20250628 07:53:48.531643 7516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
W20250628 07:53:48.535930 7516 leader_election.cc:336] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
W20250628 07:53:48.537139 7513 leader_election.cc:336] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649): Network error: Client connection negotiation failed: client connection to 127.3.91.130:33649: connect: Connection refused (error 111)
I20250628 07:53:48.537914 7513 leader_election.cc:304] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff; no voters: 84598f3817904a8794eebb4582ab6494, ce5b97082f844f69a92cb49f0407e949
I20250628 07:53:48.539127 7649 raft_consensus.cc:2747] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250628 07:53:47.671351 7643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:53:47.671363 7646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:47.673537 7630 server_base.cc:1048] running on GCE node
I20250628 07:53:48.854730 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:48.857590 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:48.859066 7630 hybrid_clock.cc:648] HybridClock initialized: now 1751097228859036 us; error 53 us; skew 500 ppm
I20250628 07:53:48.859841 7630 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:48.866215 7630 webserver.cc:469] Webserver started at http://127.3.91.130:37845/ using document root <none> and password file <none>
I20250628 07:53:48.867146 7630 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:48.867377 7630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:48.875576 7630 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250628 07:53:48.880501 7657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:48.881568 7630 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250628 07:53:48.881944 7630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce5b97082f844f69a92cb49f0407e949"
format_stamp: "Formatted at 2025-06-28 07:53:19 on dist-test-slave-wggc"
I20250628 07:53:48.883915 7630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:48.954155 7630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:48.955734 7630 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:48.956207 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:48.958793 7630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:48.959255 7649 raft_consensus.cc:491] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:53:48.959750 7649 raft_consensus.cc:513] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:48.961781 7649 leader_election.cc:290] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
W20250628 07:53:48.966307 7513 leader_election.cc:336] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649): Network error: Client connection negotiation failed: client connection to 127.3.91.130:33649: connect: Connection refused (error 111)
W20250628 07:53:48.966828 7516 leader_election.cc:336] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
I20250628 07:53:48.966923 7664 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250628 07:53:48.967206 7516 leader_election.cc:304] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff; no voters: 84598f3817904a8794eebb4582ab6494, ce5b97082f844f69a92cb49f0407e949
I20250628 07:53:48.967765 7649 raft_consensus.cc:2747] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250628 07:53:48.979779 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250628 07:53:48.980050 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.003s
I20250628 07:53:48.980358 7630 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250628 07:53:48.987417 7664 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap starting.
I20250628 07:53:48.992303 7630 ts_tablet_manager.cc:610] Registered 2 tablets
I20250628 07:53:48.992535 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.006s sys 0.004s
I20250628 07:53:49.050279 7664 log.cc:826] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Log is configured to *not* fsync() on all Append() calls
I20250628 07:53:49.164753 7630 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:33649
I20250628 07:53:49.164925 7772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:33649 every 8 connection(s)
I20250628 07:53:49.168300 7630 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:53:49.175062 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7630
I20250628 07:53:49.177927 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:35231
--local_ip_for_outbound_sockets=127.3.91.131
--tserver_master_addrs=127.3.91.190:42237
--webserver_port=46791
--webserver_interface=127.3.91.131
--builtin_ntp_servers=127.3.91.148:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250628 07:53:49.201679 7664 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:49.202831 7664 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Bootstrap complete.
I20250628 07:53:49.204631 7664 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Time spent bootstrapping tablet: real 0.218s user 0.170s sys 0.043s
I20250628 07:53:49.213842 7773 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:49.214440 7773 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:49.215777 7773 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:49.220160 7440 ts_manager.cc:194] Registered new tserver with Master: ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649)
I20250628 07:53:49.224359 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:56609
I20250628 07:53:49.222836 7664 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:49.226104 7664 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Initialized, Role: FOLLOWER
I20250628 07:53:49.227218 7664 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:49.230259 7773 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
I20250628 07:53:49.230998 7664 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949: Time spent starting tablet: real 0.026s user 0.024s sys 0.000s
I20250628 07:53:49.231745 7664 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap starting.
I20250628 07:53:49.329402 7664 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:49.330147 7664 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Bootstrap complete.
I20250628 07:53:49.331326 7664 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent bootstrapping tablet: real 0.100s user 0.090s sys 0.008s
I20250628 07:53:49.332932 7664 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:49.333395 7664 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Initialized, Role: FOLLOWER
I20250628 07:53:49.334043 7664 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:49.335363 7664 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250628 07:53:49.509783 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:53:49.510282 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:53:49.510805 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:53:49.542865 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:53:49.543803 7777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:53:49.578998 7777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:35231
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=46791
--tserver_master_addrs=127.3.91.190:42237
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:53:49.580381 7777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:53:49.582079 7777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:53:49.599673 7785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:50.643909 7791 raft_consensus.cc:491] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:53:50.644434 7791 raft_consensus.cc:513] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.646732 7791 leader_election.cc:290] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
W20250628 07:53:50.659976 7661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
W20250628 07:53:50.664520 7661 leader_election.cc:336] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
I20250628 07:53:50.669323 7582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3351b4de770f41679d0288327b32fcdf" candidate_uuid: "ce5b97082f844f69a92cb49f0407e949" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" is_pre_election: true
I20250628 07:53:50.669988 7582 raft_consensus.cc:2466] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5b97082f844f69a92cb49f0407e949 in term 2.
I20250628 07:53:50.671358 7659 leader_election.cc:304] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff, ce5b97082f844f69a92cb49f0407e949; no voters: 84598f3817904a8794eebb4582ab6494
I20250628 07:53:50.672065 7791 raft_consensus.cc:2802] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 07:53:50.672389 7791 raft_consensus.cc:491] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:53:50.672654 7791 raft_consensus.cc:3058] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:53:50.679003 7791 raft_consensus.cc:513] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.680625 7791 leader_election.cc:290] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 election: Requested vote from peers 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
I20250628 07:53:50.681492 7582 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3351b4de770f41679d0288327b32fcdf" candidate_uuid: "ce5b97082f844f69a92cb49f0407e949" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "68ea013ff1aa47f996b38a6cc264a7ff"
I20250628 07:53:50.682133 7582 raft_consensus.cc:3058] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Advancing to term 3
W20250628 07:53:50.684556 7661 leader_election.cc:336] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
I20250628 07:53:50.689042 7582 raft_consensus.cc:2466] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5b97082f844f69a92cb49f0407e949 in term 3.
I20250628 07:53:50.690016 7659 leader_election.cc:304] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff, ce5b97082f844f69a92cb49f0407e949; no voters: 84598f3817904a8794eebb4582ab6494
I20250628 07:53:50.690661 7791 raft_consensus.cc:2802] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 07:53:50.692356 7791 raft_consensus.cc:695] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 3 LEADER]: Becoming Leader. State: Replica: ce5b97082f844f69a92cb49f0407e949, State: Running, Role: LEADER
I20250628 07:53:50.693267 7791 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.705981 7798 raft_consensus.cc:491] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:53:50.706410 7798 raft_consensus.cc:513] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.706300 7440 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: term changed from 2 to 3, leader changed from <none> to ce5b97082f844f69a92cb49f0407e949 (127.3.91.130). New cstate: current_term: 3 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250628 07:53:50.708444 7798 leader_election.cc:290] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
W20250628 07:53:50.732630 7516 leader_election.cc:336] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
I20250628 07:53:50.734798 7728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e24b3f4c19d74938823c52ac93029c82" candidate_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ce5b97082f844f69a92cb49f0407e949" is_pre_election: true
I20250628 07:53:50.735473 7728 raft_consensus.cc:2466] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ea013ff1aa47f996b38a6cc264a7ff in term 2.
I20250628 07:53:50.736904 7513 leader_election.cc:304] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff, ce5b97082f844f69a92cb49f0407e949; no voters: 84598f3817904a8794eebb4582ab6494
I20250628 07:53:50.737530 7798 raft_consensus.cc:2802] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250628 07:53:50.737870 7798 raft_consensus.cc:491] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:53:50.738106 7798 raft_consensus.cc:3058] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:53:50.742743 7798 raft_consensus.cc:513] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.744280 7798 leader_election.cc:290] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 election: Requested vote from peers ce5b97082f844f69a92cb49f0407e949 (127.3.91.130:33649), 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
I20250628 07:53:50.745460 7728 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e24b3f4c19d74938823c52ac93029c82" candidate_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ce5b97082f844f69a92cb49f0407e949"
I20250628 07:53:50.746047 7728 raft_consensus.cc:3058] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 2 FOLLOWER]: Advancing to term 3
W20250628 07:53:50.748307 7516 leader_election.cc:336] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111)
I20250628 07:53:50.751312 7728 raft_consensus.cc:2466] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ea013ff1aa47f996b38a6cc264a7ff in term 3.
I20250628 07:53:50.752362 7513 leader_election.cc:304] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68ea013ff1aa47f996b38a6cc264a7ff, ce5b97082f844f69a92cb49f0407e949; no voters: 84598f3817904a8794eebb4582ab6494
I20250628 07:53:50.753002 7798 raft_consensus.cc:2802] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Leader election won for term 3
I20250628 07:53:50.755267 7798 raft_consensus.cc:695] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 LEADER]: Becoming Leader. State: Replica: 68ea013ff1aa47f996b38a6cc264a7ff, State: Running, Role: LEADER
I20250628 07:53:50.756198 7798 consensus_queue.cc:237] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:50.767941 7440 catalog_manager.cc:5582] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250628 07:53:49.600003 7788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:49.602762 7777 server_base.cc:1048] running on GCE node
W20250628 07:53:49.600555 7786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:53:50.812656 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:53:50.815105 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:53:50.816488 7777 hybrid_clock.cc:648] HybridClock initialized: now 1751097230816458 us; error 48 us; skew 500 ppm
I20250628 07:53:50.817307 7777 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:53:50.823987 7777 webserver.cc:469] Webserver started at http://127.3.91.131:46791/ using document root <none> and password file <none>
I20250628 07:53:50.824904 7777 fs_manager.cc:362] Metadata directory not provided
I20250628 07:53:50.825096 7777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:53:50.832932 7777 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20250628 07:53:50.837806 7813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:53:50.838913 7777 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250628 07:53:50.839226 7777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "84598f3817904a8794eebb4582ab6494"
format_stamp: "Formatted at 2025-06-28 07:53:21 on dist-test-slave-wggc"
I20250628 07:53:50.841132 7777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:53:50.920717 7777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:53:50.922266 7777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:53:50.922748 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:53:50.925331 7777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:53:50.931785 7820 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250628 07:53:50.947943 7777 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250628 07:53:50.948186 7777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.000s sys 0.002s
I20250628 07:53:50.948407 7777 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250628 07:53:50.953896 7820 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:50.964335 7777 ts_tablet_manager.cc:610] Registered 3 tablets
I20250628 07:53:50.964630 7777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.016s user 0.014s sys 0.000s
I20250628 07:53:51.033138 7820 log.cc:826] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Log is configured to *not* fsync() on all Append() calls
W20250628 07:53:51.127031 7661 consensus_peers.cc:489] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 -> Peer 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): Couldn't send request to peer 84598f3817904a8794eebb4582ab6494. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.131:35231: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 07:53:51.137517 7820 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:51.138700 7820 tablet_bootstrap.cc:492] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:51.140493 7820 ts_tablet_manager.cc:1397] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.187s user 0.138s sys 0.044s
I20250628 07:53:51.160727 7820 raft_consensus.cc:357] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:51.163782 7820 raft_consensus.cc:738] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: FOLLOWER
I20250628 07:53:51.164891 7820 consensus_queue.cc:260] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } } peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:51.170548 7582 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250628 07:53:51.172901 7791 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250628 07:53:51.174986 7820 ts_tablet_manager.cc:1428] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.034s user 0.018s sys 0.013s
I20250628 07:53:51.175881 7820 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:51.208134 7777 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:35231
I20250628 07:53:51.208477 7929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:35231 every 8 connection(s)
I20250628 07:53:51.212929 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7777
I20250628 07:53:51.211891 7777 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:53:51.284930 7930 heartbeater.cc:344] Connected to a master server at 127.3.91.190:42237
I20250628 07:53:51.285465 7930 heartbeater.cc:461] Registering TS with master...
I20250628 07:53:51.286756 7930 heartbeater.cc:507] Master 127.3.91.190:42237 requested a full tablet report, sending...
I20250628 07:53:51.291383 7440 ts_manager.cc:194] Registered new tserver with Master: 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231)
I20250628 07:53:51.297327 7440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:39929
I20250628 07:53:51.297205 7728 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } }
I20250628 07:53:51.302235 7582 raft_consensus.cc:1273] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Refusing update from remote peer ce5b97082f844f69a92cb49f0407e949: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250628 07:53:51.303675 7933 consensus_queue.cc:1035] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250628 07:53:51.307056 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:51.309938 7791 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:51.311841 7582 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:51.314703 7728 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P ce5b97082f844f69a92cb49f0407e949 [term 3 FOLLOWER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250628 07:53:51.316145 7798 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250628 07:53:51.317996 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:53:51.328800 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 68ea013ff1aa47f996b38a6cc264a7ff to finish bootstrapping
I20250628 07:53:51.328886 7424 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3351b4de770f41679d0288327b32fcdf with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 07:53:51.331760 7439 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff reported cstate change: config changed from index 12 to 14, VOTER 84598f3817904a8794eebb4582ab6494 (127.3.91.131) evicted. New cstate: current_term: 3 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } }
I20250628 07:53:51.362610 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce5b97082f844f69a92cb49f0407e949 to finish bootstrapping
I20250628 07:53:51.365483 7728 consensus_queue.cc:237] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } }
I20250628 07:53:51.368111 7942 raft_consensus.cc:2953] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } }
I20250628 07:53:51.376395 7883 raft_consensus.cc:3058] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:53:51.386152 7820 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:51.386384 7424 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 3351b4de770f41679d0288327b32fcdf with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 07:53:51.387173 7820 tablet_bootstrap.cc:492] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:51.388783 7820 ts_tablet_manager.cc:1397] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.213s user 0.137s sys 0.018s
I20250628 07:53:51.388747 7883 raft_consensus.cc:1273] T e24b3f4c19d74938823c52ac93029c82 P 84598f3817904a8794eebb4582ab6494 [term 3 FOLLOWER]: Refusing update from remote peer 68ea013ff1aa47f996b38a6cc264a7ff: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250628 07:53:51.391121 7945 consensus_queue.cc:1035] T e24b3f4c19d74938823c52ac93029c82 P 68ea013ff1aa47f996b38a6cc264a7ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250628 07:53:51.391166 7820 raft_consensus.cc:357] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:51.391793 7820 raft_consensus.cc:738] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: FOLLOWER
I20250628 07:53:51.392351 7820 consensus_queue.cc:260] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } } peers { permanent_uuid: "68ea013ff1aa47f996b38a6cc264a7ff" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 44329 } attrs { promote: false } } peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } attrs { promote: false } }
I20250628 07:53:51.393585 7930 heartbeater.cc:499] Master 127.3.91.190:42237 was elected leader, sending a full tablet report...
I20250628 07:53:51.395216 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 84598f3817904a8794eebb4582ab6494 to finish bootstrapping
I20250628 07:53:51.397365 7439 catalog_manager.cc:5582] T 3351b4de770f41679d0288327b32fcdf P ce5b97082f844f69a92cb49f0407e949 reported cstate change: config changed from index 14 to 15, VOTER 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129) evicted. New cstate: current_term: 3 leader_uuid: "ce5b97082f844f69a92cb49f0407e949" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "ce5b97082f844f69a92cb49f0407e949" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 33649 } health_report { overall_health: HEALTHY } } }
I20250628 07:53:51.406972 7820 ts_tablet_manager.cc:1428] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20250628 07:53:51.408313 7820 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap starting.
I20250628 07:53:51.423583 7562 tablet_service.cc:1515] Processing DeleteTablet for tablet 3351b4de770f41679d0288327b32fcdf with delete_type TABLET_DATA_TOMBSTONED (TS 68ea013ff1aa47f996b38a6cc264a7ff not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:51750
I20250628 07:53:51.428109 7953 tablet_replica.cc:331] stopping tablet replica
I20250628 07:53:51.424471 7863 tablet_service.cc:1515] Processing DeleteTablet for tablet 3351b4de770f41679d0288327b32fcdf with delete_type TABLET_DATA_TOMBSTONED (TS 84598f3817904a8794eebb4582ab6494 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:38714
I20250628 07:53:51.428768 7953 raft_consensus.cc:2241] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Raft consensus shutting down.
I20250628 07:53:51.429258 7953 raft_consensus.cc:2270] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250628 07:53:51.433462 7953 ts_tablet_manager.cc:1905] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 07:53:51.438189 7954 tablet_replica.cc:331] stopping tablet replica
I20250628 07:53:51.439702 7954 raft_consensus.cc:2241] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 07:53:51.440846 7954 raft_consensus.cc:2270] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 07:53:51.447731 7954 ts_tablet_manager.cc:1905] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 07:53:51.453154 7953 ts_tablet_manager.cc:1918] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250628 07:53:51.453733 7953 log.cc:1199] T 3351b4de770f41679d0288327b32fcdf P 68ea013ff1aa47f996b38a6cc264a7ff: Deleting WAL directory at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/wals/3351b4de770f41679d0288327b32fcdf
I20250628 07:53:51.455771 7425 catalog_manager.cc:4928] TS 68ea013ff1aa47f996b38a6cc264a7ff (127.3.91.129:44329): tablet 3351b4de770f41679d0288327b32fcdf (table TestTable [id=fb4b1a3f48224a4f8b103d900fcfaffe]) successfully deleted
I20250628 07:53:51.463379 7954 ts_tablet_manager.cc:1918] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250628 07:53:51.463732 7954 log.cc:1199] T 3351b4de770f41679d0288327b32fcdf P 84598f3817904a8794eebb4582ab6494: Deleting WAL directory at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/wals/3351b4de770f41679d0288327b32fcdf
I20250628 07:53:51.465634 7427 catalog_manager.cc:4928] TS 84598f3817904a8794eebb4582ab6494 (127.3.91.131:35231): tablet 3351b4de770f41679d0288327b32fcdf (table TestTable [id=fb4b1a3f48224a4f8b103d900fcfaffe]) successfully deleted
I20250628 07:53:51.532675 7820 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:53:51.533375 7820 tablet_bootstrap.cc:492] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Bootstrap complete.
I20250628 07:53:51.534583 7820 ts_tablet_manager.cc:1397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent bootstrapping tablet: real 0.126s user 0.084s sys 0.031s
I20250628 07:53:51.536109 7820 raft_consensus.cc:357] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:51.536478 7820 raft_consensus.cc:738] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Initialized, Role: FOLLOWER
I20250628 07:53:51.536916 7820 consensus_queue.cc:260] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:51.537297 7820 raft_consensus.cc:397] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:53:51.537513 7820 raft_consensus.cc:491] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:53:51.537878 7820 raft_consensus.cc:3058] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 2 FOLLOWER]: Advancing to term 3
I20250628 07:53:51.541306 7820 raft_consensus.cc:513] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:51.542032 7820 leader_election.cc:304] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84598f3817904a8794eebb4582ab6494; no voters:
I20250628 07:53:51.542482 7820 leader_election.cc:290] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250628 07:53:51.542740 7925 raft_consensus.cc:2802] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 3 FOLLOWER]: Leader election won for term 3
I20250628 07:53:51.544960 7925 raft_consensus.cc:695] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [term 3 LEADER]: Becoming Leader. State: Replica: 84598f3817904a8794eebb4582ab6494, State: Running, Role: LEADER
I20250628 07:53:51.545161 7820 ts_tablet_manager.cc:1428] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250628 07:53:51.545796 7925 consensus_queue.cc:237] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } }
I20250628 07:53:51.552091 7440 catalog_manager.cc:5582] T cac8fad2e5f1408880c28a9299cb3d53 P 84598f3817904a8794eebb4582ab6494 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "84598f3817904a8794eebb4582ab6494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84598f3817904a8794eebb4582ab6494" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 35231 } health_report { overall_health: HEALTHY } } }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Timed out: status_and_schema {
tablet_status {
tablet_id: "cac8fad2e5f1408880c28a9299cb3d53"
table_name: "TestTable2"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415713
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: LEADER
}
status_and_schema {
tablet_status {
tablet_id: "3351b4de770f41679d0288327b32fcdf"
table_name: "TestTable"
state: STOPPED
last_status: "Deleted tablet blocks from disk"
estimated_on_disk_size: 8401339
tablet_data_state: TABLET_DATA_TOMBSTONED
partition {
partition_key_start: ""
partition_key_end: ""
}
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: UNKNOWN_ROLE
}
status_and_schema {
tablet_status {
tablet_id: "e24b3f4c19d74938823c52ac93029c82"
table_name: "TestTable1"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415795
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: FOLLOWER
}
I20250628 07:54:11.410878 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7477
I20250628 07:54:11.437709 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7630
I20250628 07:54:11.467803 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7777
I20250628 07:54:11.498703 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7407
2025-06-28T07:54:11Z chronyd exiting
I20250628 07:54:11.548521 3438 test_util.cc:183] -----------------------------------------------
I20250628 07:54:11.548705 3438 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1751097128721223-3438-0
[ FAILED ] AdminCliTest.TestRebuildTables (57746 ms)
[----------] 5 tests from AdminCliTest (122758 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250628 07:54:11.552155 3438 test_util.cc:276] Using random seed: 430267257
I20250628 07:54:11.556164 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:54:11.556314 3438 ts_itest-base.cc:116] --------------
I20250628 07:54:11.556422 3438 ts_itest-base.cc:117] 5 tablet servers
I20250628 07:54:11.556519 3438 ts_itest-base.cc:118] 3 replicas per TS
I20250628 07:54:11.556624 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:54:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:54:11Z Disabled control of system clock
I20250628 07:54:11.600536 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:38179
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:38179
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 07:54:11.903126 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:11.903685 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:11.904101 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:11.935278 7984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:11.935721 7984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:54:11.935941 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:11.936149 7984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:54:11.936352 7984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:54:11.972059 7984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:38179
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:11.973351 7984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:11.975021 7984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:11.991773 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:11.991824 7990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:11.992846 7984 server_base.cc:1048] running on GCE node
W20250628 07:54:11.992043 7993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:13.183516 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:13.186723 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:13.188161 7984 hybrid_clock.cc:648] HybridClock initialized: now 1751097253188124 us; error 59 us; skew 500 ppm
I20250628 07:54:13.188980 7984 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:13.200591 7984 webserver.cc:469] Webserver started at http://127.3.91.190:35559/ using document root <none> and password file <none>
I20250628 07:54:13.201588 7984 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:13.201862 7984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:13.202353 7984 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:13.206923 7984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "088bfaec20624d42a70740ca8a2d14ac"
format_stamp: "Formatted at 2025-06-28 07:54:13 on dist-test-slave-wggc"
I20250628 07:54:13.208071 7984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "088bfaec20624d42a70740ca8a2d14ac"
format_stamp: "Formatted at 2025-06-28 07:54:13 on dist-test-slave-wggc"
I20250628 07:54:13.215317 7984 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20250628 07:54:13.220969 8000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:13.222034 7984 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250628 07:54:13.222363 7984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "088bfaec20624d42a70740ca8a2d14ac"
format_stamp: "Formatted at 2025-06-28 07:54:13 on dist-test-slave-wggc"
I20250628 07:54:13.222708 7984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:13.272356 7984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:13.273873 7984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:13.274331 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:13.344175 7984 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:38179
I20250628 07:54:13.344266 8051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:38179 every 8 connection(s)
I20250628 07:54:13.347074 7984 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:54:13.352203 8052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:13.353706 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 7984
I20250628 07:54:13.354238 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:54:13.376250 8052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac: Bootstrap starting.
I20250628 07:54:13.382344 8052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:13.384055 8052 log.cc:826] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:13.388813 8052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac: No bootstrap required, opened a new log
I20250628 07:54:13.406250 8052 raft_consensus.cc:357] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } }
I20250628 07:54:13.406939 8052 raft_consensus.cc:383] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:13.407192 8052 raft_consensus.cc:738] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088bfaec20624d42a70740ca8a2d14ac, State: Initialized, Role: FOLLOWER
I20250628 07:54:13.407958 8052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } }
I20250628 07:54:13.408474 8052 raft_consensus.cc:397] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:13.408740 8052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:13.409068 8052 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:13.413128 8052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } }
I20250628 07:54:13.413862 8052 leader_election.cc:304] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088bfaec20624d42a70740ca8a2d14ac; no voters:
I20250628 07:54:13.415508 8052 leader_election.cc:290] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:54:13.416234 8057 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:13.418884 8057 raft_consensus.cc:695] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [term 1 LEADER]: Becoming Leader. State: Replica: 088bfaec20624d42a70740ca8a2d14ac, State: Running, Role: LEADER
I20250628 07:54:13.419516 8052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:54:13.419674 8057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } }
I20250628 07:54:13.431531 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088bfaec20624d42a70740ca8a2d14ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } } }
I20250628 07:54:13.431772 8059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088bfaec20624d42a70740ca8a2d14ac. Latest consensus state: current_term: 1 leader_uuid: "088bfaec20624d42a70740ca8a2d14ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088bfaec20624d42a70740ca8a2d14ac" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 38179 } } }
I20250628 07:54:13.432411 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:13.432507 8059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:13.435612 8066 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:54:13.448320 8066 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:54:13.465029 8066 catalog_manager.cc:1349] Generated new cluster ID: 3af6837143994d238cb76345cde9531e
I20250628 07:54:13.465351 8066 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:54:13.478018 8066 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:54:13.479470 8066 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:54:13.493474 8066 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 088bfaec20624d42a70740ca8a2d14ac: Generated new TSK 0
I20250628 07:54:13.494475 8066 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:54:13.505015 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250628 07:54:13.812052 8076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:13.812587 8076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:13.813097 8076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:13.844735 8076 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:13.845139 8076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:13.845981 8076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:54:13.881108 8076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:13.882462 8076 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:13.884027 8076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:13.901578 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:13.902282 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:13.903825 8076 server_base.cc:1048] running on GCE node
W20250628 07:54:13.903565 8085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:15.076738 8076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:15.079789 8076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:15.081265 8076 hybrid_clock.cc:648] HybridClock initialized: now 1751097255081206 us; error 78 us; skew 500 ppm
I20250628 07:54:15.082111 8076 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:15.088780 8076 webserver.cc:469] Webserver started at http://127.3.91.129:37861/ using document root <none> and password file <none>
I20250628 07:54:15.089740 8076 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:15.089974 8076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:15.090423 8076 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:15.094823 8076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cbdcdcc47b104989b3b2331059ce201f"
format_stamp: "Formatted at 2025-06-28 07:54:15 on dist-test-slave-wggc"
I20250628 07:54:15.095888 8076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cbdcdcc47b104989b3b2331059ce201f"
format_stamp: "Formatted at 2025-06-28 07:54:15 on dist-test-slave-wggc"
I20250628 07:54:15.103096 8076 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250628 07:54:15.109097 8092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:15.110201 8076 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250628 07:54:15.110507 8076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cbdcdcc47b104989b3b2331059ce201f"
format_stamp: "Formatted at 2025-06-28 07:54:15 on dist-test-slave-wggc"
I20250628 07:54:15.110811 8076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:15.174316 8076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:15.175776 8076 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:15.176201 8076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:15.178757 8076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:15.182773 8076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:15.182972 8076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:15.183239 8076 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:15.183390 8076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:15.318846 8076 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:43963
I20250628 07:54:15.318925 8204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:43963 every 8 connection(s)
I20250628 07:54:15.321476 8076 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:54:15.325695 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8076
I20250628 07:54:15.326233 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:54:15.332865 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 07:54:15.342860 8205 heartbeater.cc:344] Connected to a master server at 127.3.91.190:38179
I20250628 07:54:15.343284 8205 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:15.344345 8205 heartbeater.cc:507] Master 127.3.91.190:38179 requested a full tablet report, sending...
I20250628 07:54:15.346884 8017 ts_manager.cc:194] Registered new tserver with Master: cbdcdcc47b104989b3b2331059ce201f (127.3.91.129:43963)
I20250628 07:54:15.348778 8017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:53709
W20250628 07:54:15.651221 8209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:15.651718 8209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:15.652318 8209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:15.684329 8209 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:15.684695 8209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:15.685415 8209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:54:15.719892 8209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:15.721210 8209 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:15.722810 8209 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250628 07:54:16.352077 8205 heartbeater.cc:499] Master 127.3.91.190:38179 was elected leader, sending a full tablet report...
W20250628 07:54:15.740252 8218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:15.738945 8216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:15.740975 8209 server_base.cc:1048] running on GCE node
W20250628 07:54:15.738930 8215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:16.925969 8209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:16.928208 8209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:16.929574 8209 hybrid_clock.cc:648] HybridClock initialized: now 1751097256929540 us; error 42 us; skew 500 ppm
I20250628 07:54:16.930397 8209 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:16.937126 8209 webserver.cc:469] Webserver started at http://127.3.91.130:45643/ using document root <none> and password file <none>
I20250628 07:54:16.938180 8209 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:16.938414 8209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:16.938864 8209 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:16.943359 8209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ccd382e5568d4a618f3ac653ea8d1b29"
format_stamp: "Formatted at 2025-06-28 07:54:16 on dist-test-slave-wggc"
I20250628 07:54:16.944432 8209 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ccd382e5568d4a618f3ac653ea8d1b29"
format_stamp: "Formatted at 2025-06-28 07:54:16 on dist-test-slave-wggc"
I20250628 07:54:16.951561 8209 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250628 07:54:16.957270 8225 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:16.958534 8209 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:54:16.958859 8209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ccd382e5568d4a618f3ac653ea8d1b29"
format_stamp: "Formatted at 2025-06-28 07:54:16 on dist-test-slave-wggc"
I20250628 07:54:16.959225 8209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:17.008538 8209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:17.010059 8209 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:17.010511 8209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:17.012992 8209 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:17.017078 8209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:17.017298 8209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:17.017544 8209 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:17.017763 8209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250628 07:54:17.153139 8209 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:44239
I20250628 07:54:17.153260 8337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:44239 every 8 connection(s)
I20250628 07:54:17.155828 8209 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250628 07:54:17.163110 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8209
I20250628 07:54:17.163645 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250628 07:54:17.171233 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 07:54:17.179877 8338 heartbeater.cc:344] Connected to a master server at 127.3.91.190:38179
I20250628 07:54:17.180449 8338 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:17.181865 8338 heartbeater.cc:507] Master 127.3.91.190:38179 requested a full tablet report, sending...
I20250628 07:54:17.184298 8017 ts_manager.cc:194] Registered new tserver with Master: ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239)
I20250628 07:54:17.186180 8017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:42141
W20250628 07:54:17.481297 8342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:17.481796 8342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:17.482255 8342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:17.513533 8342 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:17.513926 8342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:17.514662 8342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:54:17.549223 8342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:17.550526 8342 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:17.552073 8342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:17.568617 8349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:18.189563 8338 heartbeater.cc:499] Master 127.3.91.190:38179 was elected leader, sending a full tablet report...
W20250628 07:54:17.568660 8351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:17.570765 8342 server_base.cc:1048] running on GCE node
W20250628 07:54:17.568606 8348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:18.756474 8342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:18.758919 8342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:18.760351 8342 hybrid_clock.cc:648] HybridClock initialized: now 1751097258760281 us; error 75 us; skew 500 ppm
I20250628 07:54:18.761231 8342 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:18.772750 8342 webserver.cc:469] Webserver started at http://127.3.91.131:38499/ using document root <none> and password file <none>
I20250628 07:54:18.773804 8342 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:18.774027 8342 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:18.774503 8342 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:18.779176 8342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7fa5bae76da7475a8d9fe80364bf79fc"
format_stamp: "Formatted at 2025-06-28 07:54:18 on dist-test-slave-wggc"
I20250628 07:54:18.780315 8342 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7fa5bae76da7475a8d9fe80364bf79fc"
format_stamp: "Formatted at 2025-06-28 07:54:18 on dist-test-slave-wggc"
I20250628 07:54:18.787590 8342 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20250628 07:54:18.793507 8358 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:18.794649 8342 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250628 07:54:18.794979 8342 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7fa5bae76da7475a8d9fe80364bf79fc"
format_stamp: "Formatted at 2025-06-28 07:54:18 on dist-test-slave-wggc"
I20250628 07:54:18.795327 8342 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:18.843087 8342 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:18.844568 8342 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:18.845009 8342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:18.847607 8342 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:18.851645 8342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:18.851845 8342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:18.852114 8342 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:18.852273 8342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:18.990046 8342 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:41813
I20250628 07:54:18.990124 8470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:41813 every 8 connection(s)
I20250628 07:54:18.992609 8342 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250628 07:54:18.995805 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8342
I20250628 07:54:18.996433 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250628 07:54:19.003306 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.132:0
--local_ip_for_outbound_sockets=127.3.91.132
--webserver_interface=127.3.91.132
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 07:54:19.014562 8471 heartbeater.cc:344] Connected to a master server at 127.3.91.190:38179
I20250628 07:54:19.015118 8471 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:19.016502 8471 heartbeater.cc:507] Master 127.3.91.190:38179 requested a full tablet report, sending...
I20250628 07:54:19.019127 8017 ts_manager.cc:194] Registered new tserver with Master: 7fa5bae76da7475a8d9fe80364bf79fc (127.3.91.131:41813)
I20250628 07:54:19.020402 8017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:60839
W20250628 07:54:19.316807 8475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:19.317472 8475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:19.318092 8475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:19.349582 8475 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:19.350137 8475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:19.350936 8475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.132
I20250628 07:54:19.387197 8475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.3.91.132
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.132
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:19.388521 8475 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:19.390187 8475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:19.407528 8483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:20.024303 8471 heartbeater.cc:499] Master 127.3.91.190:38179 was elected leader, sending a full tablet report...
W20250628 07:54:19.407712 8485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:19.407550 8482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:20.597426 8484 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:54:20.597573 8475 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:54:20.601850 8475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:20.604604 8475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:20.606058 8475 hybrid_clock.cc:648] HybridClock initialized: now 1751097260606019 us; error 61 us; skew 500 ppm
I20250628 07:54:20.606914 8475 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:20.613204 8475 webserver.cc:469] Webserver started at http://127.3.91.132:39913/ using document root <none> and password file <none>
I20250628 07:54:20.614193 8475 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:20.614408 8475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:20.614881 8475 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:20.619458 8475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6debfc31fb33439c8285a04a8fd6a0ba"
format_stamp: "Formatted at 2025-06-28 07:54:20 on dist-test-slave-wggc"
I20250628 07:54:20.620594 8475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6debfc31fb33439c8285a04a8fd6a0ba"
format_stamp: "Formatted at 2025-06-28 07:54:20 on dist-test-slave-wggc"
I20250628 07:54:20.627664 8475 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250628 07:54:20.633288 8492 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:20.634377 8475 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250628 07:54:20.634701 8475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6debfc31fb33439c8285a04a8fd6a0ba"
format_stamp: "Formatted at 2025-06-28 07:54:20 on dist-test-slave-wggc"
I20250628 07:54:20.635049 8475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:20.696623 8475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:20.698096 8475 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:20.698529 8475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:20.700953 8475 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:20.704941 8475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:20.705137 8475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:20.705330 8475 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:20.705464 8475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:20.839848 8475 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.132:34463
I20250628 07:54:20.839956 8604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.132:34463 every 8 connection(s)
I20250628 07:54:20.842430 8475 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250628 07:54:20.847211 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8475
I20250628 07:54:20.847993 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250628 07:54:20.854760 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.133:0
--local_ip_for_outbound_sockets=127.3.91.133
--webserver_interface=127.3.91.133
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--builtin_ntp_servers=127.3.91.148:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250628 07:54:20.864143 8605 heartbeater.cc:344] Connected to a master server at 127.3.91.190:38179
I20250628 07:54:20.864658 8605 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:20.865743 8605 heartbeater.cc:507] Master 127.3.91.190:38179 requested a full tablet report, sending...
I20250628 07:54:20.867874 8017 ts_manager.cc:194] Registered new tserver with Master: 6debfc31fb33439c8285a04a8fd6a0ba (127.3.91.132:34463)
I20250628 07:54:20.869161 8017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.132:43767
W20250628 07:54:21.165198 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:21.165750 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:21.166339 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:21.199044 8609 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250628 07:54:21.199483 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:21.200269 8609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.133
I20250628 07:54:21.235268 8609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.3.91.133
--webserver_port=0
--tserver_master_addrs=127.3.91.190:38179
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.133
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:21.236658 8609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:21.238459 8609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:21.255369 8616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:21.872337 8605 heartbeater.cc:499] Master 127.3.91.190:38179 was elected leader, sending a full tablet report...
W20250628 07:54:21.256004 8615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:21.257992 8609 server_base.cc:1048] running on GCE node
W20250628 07:54:21.256363 8618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:22.436626 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:22.438972 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:22.440341 8609 hybrid_clock.cc:648] HybridClock initialized: now 1751097262440294 us; error 57 us; skew 500 ppm
I20250628 07:54:22.441187 8609 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:22.448379 8609 webserver.cc:469] Webserver started at http://127.3.91.133:43165/ using document root <none> and password file <none>
I20250628 07:54:22.449309 8609 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:22.449532 8609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:22.450031 8609 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:22.454780 8609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "cd2af467325646b7a6bf29cd2fe4d84a"
format_stamp: "Formatted at 2025-06-28 07:54:22 on dist-test-slave-wggc"
I20250628 07:54:22.455860 8609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "cd2af467325646b7a6bf29cd2fe4d84a"
format_stamp: "Formatted at 2025-06-28 07:54:22 on dist-test-slave-wggc"
I20250628 07:54:22.463049 8609 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 07:54:22.468636 8626 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:22.469779 8609 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250628 07:54:22.470073 8609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "cd2af467325646b7a6bf29cd2fe4d84a"
format_stamp: "Formatted at 2025-06-28 07:54:22 on dist-test-slave-wggc"
I20250628 07:54:22.470382 8609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:22.522944 8609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:22.524462 8609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:22.524876 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:22.527330 8609 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:22.531664 8609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:22.531872 8609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:22.532064 8609 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:22.532197 8609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:22.669251 8609 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.133:44251
I20250628 07:54:22.669327 8738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.133:44251 every 8 connection(s)
I20250628 07:54:22.671821 8609 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250628 07:54:22.677577 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8609
I20250628 07:54:22.678113 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250628 07:54:22.692932 8739 heartbeater.cc:344] Connected to a master server at 127.3.91.190:38179
I20250628 07:54:22.693325 8739 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:22.694343 8739 heartbeater.cc:507] Master 127.3.91.190:38179 requested a full tablet report, sending...
I20250628 07:54:22.696477 8017 ts_manager.cc:194] Registered new tserver with Master: cd2af467325646b7a6bf29cd2fe4d84a (127.3.91.133:44251)
I20250628 07:54:22.697925 8017 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.133:47195
I20250628 07:54:22.698652 3438 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 07:54:22.736012 8017 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40832:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 07:54:22.819020 8140 tablet_service.cc:1468] Processing CreateTablet for tablet e1504dc2f71d48c2aa083cb49df2f517 (DEFAULT_TABLE table=TestTable [id=9291be619f6d4225aeb138c592b299e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:22.818977 8273 tablet_service.cc:1468] Processing CreateTablet for tablet e1504dc2f71d48c2aa083cb49df2f517 (DEFAULT_TABLE table=TestTable [id=9291be619f6d4225aeb138c592b299e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:22.820963 8273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1504dc2f71d48c2aa083cb49df2f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:22.821071 8140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1504dc2f71d48c2aa083cb49df2f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:22.833938 8406 tablet_service.cc:1468] Processing CreateTablet for tablet e1504dc2f71d48c2aa083cb49df2f517 (DEFAULT_TABLE table=TestTable [id=9291be619f6d4225aeb138c592b299e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:22.835482 8406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1504dc2f71d48c2aa083cb49df2f517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:22.852977 8759 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Bootstrap starting.
I20250628 07:54:22.863111 8758 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Bootstrap starting.
I20250628 07:54:22.865028 8759 tablet_bootstrap.cc:654] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:22.867578 8759 log.cc:826] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:22.871150 8760 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Bootstrap starting.
I20250628 07:54:22.871310 8758 tablet_bootstrap.cc:654] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:22.873183 8759 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: No bootstrap required, opened a new log
I20250628 07:54:22.873638 8759 ts_tablet_manager.cc:1397] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20250628 07:54:22.873672 8758 log.cc:826] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:22.880102 8758 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: No bootstrap required, opened a new log
I20250628 07:54:22.880648 8760 tablet_bootstrap.cc:654] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:22.880613 8758 ts_tablet_manager.cc:1397] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250628 07:54:22.883018 8760 log.cc:826] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:22.888638 8760 tablet_bootstrap.cc:492] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: No bootstrap required, opened a new log
I20250628 07:54:22.889154 8760 ts_tablet_manager.cc:1397] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20250628 07:54:22.894707 8759 raft_consensus.cc:357] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.895615 8759 raft_consensus.cc:383] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:22.895923 8759 raft_consensus.cc:738] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd382e5568d4a618f3ac653ea8d1b29, State: Initialized, Role: FOLLOWER
I20250628 07:54:22.897168 8759 consensus_queue.cc:260] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.901016 8759 ts_tablet_manager.cc:1428] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Time spent starting tablet: real 0.027s user 0.022s sys 0.004s
I20250628 07:54:22.905822 8758 raft_consensus.cc:357] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.906487 8758 raft_consensus.cc:383] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:22.906749 8758 raft_consensus.cc:738] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdcdcc47b104989b3b2331059ce201f, State: Initialized, Role: FOLLOWER
I20250628 07:54:22.907907 8758 consensus_queue.cc:260] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.911880 8758 ts_tablet_manager.cc:1428] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Time spent starting tablet: real 0.031s user 0.027s sys 0.004s
W20250628 07:54:22.917418 8339 tablet.cc:2378] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:54:22.917382 8760 raft_consensus.cc:357] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.918170 8760 raft_consensus.cc:383] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:22.918516 8760 raft_consensus.cc:738] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa5bae76da7475a8d9fe80364bf79fc, State: Initialized, Role: FOLLOWER
I20250628 07:54:22.919366 8760 consensus_queue.cc:260] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:22.923363 8760 ts_tablet_manager.cc:1428] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Time spent starting tablet: real 0.034s user 0.030s sys 0.004s
W20250628 07:54:23.002020 8472 tablet.cc:2378] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:54:23.084398 8206 tablet.cc:2378] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:54:23.131930 8764 raft_consensus.cc:491] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:54:23.132455 8764 raft_consensus.cc:513] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:23.133308 8765 raft_consensus.cc:491] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:54:23.133843 8765 raft_consensus.cc:513] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:23.135519 8764 leader_election.cc:290] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbdcdcc47b104989b3b2331059ce201f (127.3.91.129:43963), 7fa5bae76da7475a8d9fe80364bf79fc (127.3.91.131:41813)
I20250628 07:54:23.139820 8765 leader_election.cc:290] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239), 7fa5bae76da7475a8d9fe80364bf79fc (127.3.91.131:41813)
I20250628 07:54:23.154577 8293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "cbdcdcc47b104989b3b2331059ce201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" is_pre_election: true
I20250628 07:54:23.155378 8293 raft_consensus.cc:2466] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdcdcc47b104989b3b2331059ce201f in term 0.
I20250628 07:54:23.156692 8160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdcdcc47b104989b3b2331059ce201f" is_pre_election: true
I20250628 07:54:23.157419 8160 raft_consensus.cc:2466] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccd382e5568d4a618f3ac653ea8d1b29 in term 0.
I20250628 07:54:23.158612 8094 leader_election.cc:304] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbdcdcc47b104989b3b2331059ce201f, ccd382e5568d4a618f3ac653ea8d1b29; no voters:
I20250628 07:54:23.159396 8765 raft_consensus.cc:2802] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:54:23.159731 8765 raft_consensus.cc:491] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:54:23.159948 8226 leader_election.cc:304] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbdcdcc47b104989b3b2331059ce201f, ccd382e5568d4a618f3ac653ea8d1b29; no voters:
I20250628 07:54:23.160121 8765 raft_consensus.cc:3058] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:23.160141 8426 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "cbdcdcc47b104989b3b2331059ce201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" is_pre_election: true
I20250628 07:54:23.160645 8764 raft_consensus.cc:2802] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:54:23.161022 8764 raft_consensus.cc:491] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:54:23.161029 8426 raft_consensus.cc:2466] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbdcdcc47b104989b3b2331059ce201f in term 0.
I20250628 07:54:23.161338 8764 raft_consensus.cc:3058] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:23.163496 8426 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" is_pre_election: true
I20250628 07:54:23.164096 8426 raft_consensus.cc:2466] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccd382e5568d4a618f3ac653ea8d1b29 in term 0.
I20250628 07:54:23.167227 8765 raft_consensus.cc:513] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:23.167955 8764 raft_consensus.cc:513] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:23.168764 8765 leader_election.cc:290] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [CANDIDATE]: Term 1 election: Requested vote from peers ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239), 7fa5bae76da7475a8d9fe80364bf79fc (127.3.91.131:41813)
I20250628 07:54:23.169792 8293 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "cbdcdcc47b104989b3b2331059ce201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccd382e5568d4a618f3ac653ea8d1b29"
I20250628 07:54:23.169965 8426 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "cbdcdcc47b104989b3b2331059ce201f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa5bae76da7475a8d9fe80364bf79fc"
I20250628 07:54:23.170434 8426 raft_consensus.cc:3058] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:23.170593 8293 raft_consensus.cc:2391] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cbdcdcc47b104989b3b2331059ce201f in current term 1: Already voted for candidate ccd382e5568d4a618f3ac653ea8d1b29 in this term.
I20250628 07:54:23.171077 8160 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdcdcc47b104989b3b2331059ce201f"
I20250628 07:54:23.171175 8764 leader_election.cc:290] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [CANDIDATE]: Term 1 election: Requested vote from peers cbdcdcc47b104989b3b2331059ce201f (127.3.91.129:43963), 7fa5bae76da7475a8d9fe80364bf79fc (127.3.91.131:41813)
I20250628 07:54:23.171844 8160 raft_consensus.cc:2391] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ccd382e5568d4a618f3ac653ea8d1b29 in current term 1: Already voted for candidate cbdcdcc47b104989b3b2331059ce201f in this term.
I20250628 07:54:23.173084 8425 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e1504dc2f71d48c2aa083cb49df2f517" candidate_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fa5bae76da7475a8d9fe80364bf79fc"
I20250628 07:54:23.174558 8228 leader_election.cc:304] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccd382e5568d4a618f3ac653ea8d1b29; no voters: 7fa5bae76da7475a8d9fe80364bf79fc, cbdcdcc47b104989b3b2331059ce201f
I20250628 07:54:23.175204 8764 raft_consensus.cc:2747] T e1504dc2f71d48c2aa083cb49df2f517 P ccd382e5568d4a618f3ac653ea8d1b29 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250628 07:54:23.178759 8426 raft_consensus.cc:2466] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbdcdcc47b104989b3b2331059ce201f in term 1.
I20250628 07:54:23.179816 8095 leader_election.cc:304] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7fa5bae76da7475a8d9fe80364bf79fc, cbdcdcc47b104989b3b2331059ce201f; no voters: ccd382e5568d4a618f3ac653ea8d1b29
I20250628 07:54:23.180570 8765 raft_consensus.cc:2802] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:23.182634 8765 raft_consensus.cc:695] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [term 1 LEADER]: Becoming Leader. State: Replica: cbdcdcc47b104989b3b2331059ce201f, State: Running, Role: LEADER
I20250628 07:54:23.183460 8765 consensus_queue.cc:237] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } }
I20250628 07:54:23.196542 8016 catalog_manager.cc:5582] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f reported cstate change: term changed from 0 to 1, leader changed from <none> to cbdcdcc47b104989b3b2331059ce201f (127.3.91.129). New cstate: current_term: 1 leader_uuid: "cbdcdcc47b104989b3b2331059ce201f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbdcdcc47b104989b3b2331059ce201f" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 43963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccd382e5568d4a618f3ac653ea8d1b29" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 } health_report { overall_health: UNKNOWN } } }
I20250628 07:54:23.215076 3438 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250628 07:54:23.218818 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cbdcdcc47b104989b3b2331059ce201f to finish bootstrapping
I20250628 07:54:23.232118 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ccd382e5568d4a618f3ac653ea8d1b29 to finish bootstrapping
I20250628 07:54:23.243280 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7fa5bae76da7475a8d9fe80364bf79fc to finish bootstrapping
I20250628 07:54:23.254184 3438 test_util.cc:276] Using random seed: 441969297
I20250628 07:54:23.277550 3438 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250628 07:54:23.278549 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8209
I20250628 07:54:23.311611 8426 raft_consensus.cc:1273] T e1504dc2f71d48c2aa083cb49df2f517 P 7fa5bae76da7475a8d9fe80364bf79fc [term 1 FOLLOWER]: Refusing update from remote peer cbdcdcc47b104989b3b2331059ce201f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250628 07:54:23.311929 8094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.130:44239: connect: Connection refused (error 111)
I20250628 07:54:23.313338 8772 consensus_queue.cc:1035] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fa5bae76da7475a8d9fe80364bf79fc" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 41813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250628 07:54:23.322037 8094 consensus_peers.cc:489] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f -> Peer ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239): Couldn't send request to peer ccd382e5568d4a618f3ac653ea8d1b29. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.130:44239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 07:54:23.337960 8783 mvcc.cc:204] Tried to move back new op lower bound from 7172494390508285952 to 7172494390007357440. Current Snapshot: MvccSnapshot[applied={T|T < 7172494390508285952}]
I20250628 07:54:23.341635 8786 mvcc.cc:204] Tried to move back new op lower bound from 7172494390508285952 to 7172494390007357440. Current Snapshot: MvccSnapshot[applied={T|T < 7172494390508285952}]
I20250628 07:54:23.701857 8739 heartbeater.cc:499] Master 127.3.91.190:38179 was elected leader, sending a full tablet report...
W20250628 07:54:25.769395 8601 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2ee0 after lost signal to thread 8477
W20250628 07:54:25.770426 8601 debug-util.cc:398] Leaking SignalData structure 0x7b08000b25a0 after lost signal to thread 8604
W20250628 07:54:26.019618 8094 consensus_peers.cc:489] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f -> Peer ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239): Couldn't send request to peer ccd382e5568d4a618f3ac653ea8d1b29. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.130:44239: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250628 07:54:26.212960 8406 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:54:26.257886 8140 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:54:26.306057 8674 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:54:26.306052 8540 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:54:28.219173 8540 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:54:28.239847 8140 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:54:28.250739 8406 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:54:28.271890 8674 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250628 07:54:28.586810 8094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.91.130:44239: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250628 07:54:28.611917 8094 consensus_peers.cc:489] T e1504dc2f71d48c2aa083cb49df2f517 P cbdcdcc47b104989b3b2331059ce201f -> Peer ccd382e5568d4a618f3ac653ea8d1b29 (127.3.91.130:44239): Couldn't send request to peer ccd382e5568d4a618f3ac653ea8d1b29. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.130:44239: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250628 07:54:29.779556 8601 debug-util.cc:398] Leaking SignalData structure 0x7b08000362e0 after lost signal to thread 8477
W20250628 07:54:29.781000 8601 debug-util.cc:398] Leaking SignalData structure 0x7b08000aa200 after lost signal to thread 8604
I20250628 07:54:30.718559 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8076
I20250628 07:54:30.764205 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8342
I20250628 07:54:30.801682 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8475
I20250628 07:54:30.828038 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8609
I20250628 07:54:30.853279 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 7984
2025-06-28T07:54:30Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19362 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19363 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250628 07:54:30.915678 3438 test_util.cc:276] Using random seed: 449630783
I20250628 07:54:30.919883 3438 ts_itest-base.cc:115] Starting cluster with:
I20250628 07:54:30.920058 3438 ts_itest-base.cc:116] --------------
I20250628 07:54:30.920218 3438 ts_itest-base.cc:117] 1 tablet servers
I20250628 07:54:30.920388 3438 ts_itest-base.cc:118] 1 replicas per TS
I20250628 07:54:30.920575 3438 ts_itest-base.cc:119] --------------
2025-06-28T07:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:54:30Z Disabled control of system clock
I20250628 07:54:30.965509 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43275
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:43275 with env {}
W20250628 07:54:31.267799 8890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:31.268411 8890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:31.268895 8890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:31.300323 8890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:54:31.300675 8890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:31.300943 8890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:54:31.301190 8890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:54:31.336555 8890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:43275
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:43275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:31.337950 8890 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:31.339663 8890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:31.356184 8899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:31.356386 8897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:31.356617 8896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:31.357118 8890 server_base.cc:1048] running on GCE node
I20250628 07:54:32.550535 8890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:32.553589 8890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:32.555056 8890 hybrid_clock.cc:648] HybridClock initialized: now 1751097272555000 us; error 73 us; skew 500 ppm
I20250628 07:54:32.555812 8890 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:32.568166 8890 webserver.cc:469] Webserver started at http://127.3.91.190:41363/ using document root <none> and password file <none>
I20250628 07:54:32.569118 8890 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:32.569314 8890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:32.569789 8890 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:32.574370 8890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "164a2c9b953246ee8df8caf051688397"
format_stamp: "Formatted at 2025-06-28 07:54:32 on dist-test-slave-wggc"
I20250628 07:54:32.575654 8890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "164a2c9b953246ee8df8caf051688397"
format_stamp: "Formatted at 2025-06-28 07:54:32 on dist-test-slave-wggc"
I20250628 07:54:32.582846 8890 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250628 07:54:32.588179 8906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:32.589146 8890 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250628 07:54:32.589421 8890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
uuid: "164a2c9b953246ee8df8caf051688397"
format_stamp: "Formatted at 2025-06-28 07:54:32 on dist-test-slave-wggc"
I20250628 07:54:32.589756 8890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:32.642333 8890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:32.643759 8890 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:32.644167 8890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:32.712894 8890 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:43275
I20250628 07:54:32.712996 8957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:43275 every 8 connection(s)
I20250628 07:54:32.715476 8890 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250628 07:54:32.720250 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8890
I20250628 07:54:32.720501 8958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:32.720649 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250628 07:54:32.742556 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397: Bootstrap starting.
I20250628 07:54:32.747875 8958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:32.749641 8958 log.cc:826] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:32.754228 8958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397: No bootstrap required, opened a new log
I20250628 07:54:32.771301 8958 raft_consensus.cc:357] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } }
I20250628 07:54:32.771947 8958 raft_consensus.cc:383] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:32.772208 8958 raft_consensus.cc:738] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164a2c9b953246ee8df8caf051688397, State: Initialized, Role: FOLLOWER
I20250628 07:54:32.772881 8958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } }
I20250628 07:54:32.773378 8958 raft_consensus.cc:397] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:32.773663 8958 raft_consensus.cc:491] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:32.773988 8958 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:32.778183 8958 raft_consensus.cc:513] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } }
I20250628 07:54:32.778843 8958 leader_election.cc:304] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164a2c9b953246ee8df8caf051688397; no voters:
I20250628 07:54:32.780472 8958 leader_election.cc:290] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:54:32.781158 8963 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:32.783463 8963 raft_consensus.cc:695] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [term 1 LEADER]: Becoming Leader. State: Replica: 164a2c9b953246ee8df8caf051688397, State: Running, Role: LEADER
I20250628 07:54:32.784204 8963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } }
I20250628 07:54:32.785161 8958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:54:32.795558 8965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164a2c9b953246ee8df8caf051688397. Latest consensus state: current_term: 1 leader_uuid: "164a2c9b953246ee8df8caf051688397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } } }
I20250628 07:54:32.796249 8965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:32.799059 8964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164a2c9b953246ee8df8caf051688397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164a2c9b953246ee8df8caf051688397" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 43275 } } }
I20250628 07:54:32.799553 8973 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:54:32.799849 8964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:32.811259 8973 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:54:32.828310 8973 catalog_manager.cc:1349] Generated new cluster ID: 0c4085d75d49431688cabb340eb04e51
I20250628 07:54:32.828568 8973 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:54:32.852757 8973 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:54:32.854203 8973 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:54:32.874264 8973 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 164a2c9b953246ee8df8caf051688397: Generated new TSK 0
I20250628 07:54:32.875385 8973 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:54:32.895547 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43275
--builtin_ntp_servers=127.3.91.148:41261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250628 07:54:33.197041 8982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:33.197649 8982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:33.198160 8982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:33.235005 8982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:33.236423 8982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:54:33.271615 8982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:41261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:43275
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:33.272915 8982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:33.274672 8982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:33.292565 8991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:33.295749 8988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:33.292831 8989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:34.651860 8990 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1356 milliseconds
I20250628 07:54:34.652014 8982 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:54:34.653542 8982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:34.656396 8982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:34.657925 8982 hybrid_clock.cc:648] HybridClock initialized: now 1751097274657873 us; error 55 us; skew 500 ppm
I20250628 07:54:34.658998 8982 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:34.666604 8982 webserver.cc:469] Webserver started at http://127.3.91.129:43645/ using document root <none> and password file <none>
I20250628 07:54:34.667829 8982 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:34.668097 8982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:34.668676 8982 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:34.675099 8982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7c82736b81884d95a2a4d3af10daa780"
format_stamp: "Formatted at 2025-06-28 07:54:34 on dist-test-slave-wggc"
I20250628 07:54:34.676577 8982 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7c82736b81884d95a2a4d3af10daa780"
format_stamp: "Formatted at 2025-06-28 07:54:34 on dist-test-slave-wggc"
I20250628 07:54:34.685379 8982 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.005s
I20250628 07:54:34.691648 8998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:34.692756 8982 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 07:54:34.693070 8982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7c82736b81884d95a2a4d3af10daa780"
format_stamp: "Formatted at 2025-06-28 07:54:34 on dist-test-slave-wggc"
I20250628 07:54:34.693401 8982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:34.763231 8982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:34.765059 8982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:34.765475 8982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:34.768517 8982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:34.772737 8982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:34.772943 8982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:34.773185 8982 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:34.773341 8982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:34.945482 8982 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:41739
I20250628 07:54:34.945587 9110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:41739 every 8 connection(s)
I20250628 07:54:34.948076 8982 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250628 07:54:34.952476 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 8982
I20250628 07:54:34.952953 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1751097128721223-3438-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250628 07:54:34.974656 9111 heartbeater.cc:344] Connected to a master server at 127.3.91.190:43275
I20250628 07:54:34.975090 9111 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:34.976048 9111 heartbeater.cc:507] Master 127.3.91.190:43275 requested a full tablet report, sending...
I20250628 07:54:34.978492 8923 ts_manager.cc:194] Registered new tserver with Master: 7c82736b81884d95a2a4d3af10daa780 (127.3.91.129:41739)
I20250628 07:54:34.980476 8923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:53747
I20250628 07:54:34.988143 3438 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 07:54:35.019953 8923 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60482:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250628 07:54:35.084311 9046 tablet_service.cc:1468] Processing CreateTablet for tablet a5dbf8dde5a74176875e2fe92537a237 (DEFAULT_TABLE table=TestTable [id=58b91d030bc644d3818362dd72787e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:35.086267 9046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5dbf8dde5a74176875e2fe92537a237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:35.114734 9126 tablet_bootstrap.cc:492] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: Bootstrap starting.
I20250628 07:54:35.120445 9126 tablet_bootstrap.cc:654] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:35.122282 9126 log.cc:826] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:35.128458 9126 tablet_bootstrap.cc:492] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: No bootstrap required, opened a new log
I20250628 07:54:35.129048 9126 ts_tablet_manager.cc:1397] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20250628 07:54:35.155655 9126 raft_consensus.cc:357] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82736b81884d95a2a4d3af10daa780" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 41739 } }
I20250628 07:54:35.156411 9126 raft_consensus.cc:383] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:35.156701 9126 raft_consensus.cc:738] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c82736b81884d95a2a4d3af10daa780, State: Initialized, Role: FOLLOWER
I20250628 07:54:35.157624 9126 consensus_queue.cc:260] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82736b81884d95a2a4d3af10daa780" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 41739 } }
I20250628 07:54:35.158289 9126 raft_consensus.cc:397] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:35.158617 9126 raft_consensus.cc:491] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:35.159015 9126 raft_consensus.cc:3058] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:35.165376 9126 raft_consensus.cc:513] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82736b81884d95a2a4d3af10daa780" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 41739 } }
I20250628 07:54:35.166291 9126 leader_election.cc:304] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c82736b81884d95a2a4d3af10daa780; no voters:
I20250628 07:54:35.168372 9126 leader_election.cc:290] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:54:35.168723 9128 raft_consensus.cc:2802] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:35.171967 9111 heartbeater.cc:499] Master 127.3.91.190:43275 was elected leader, sending a full tablet report...
I20250628 07:54:35.172763 9126 ts_tablet_manager.cc:1428] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780: Time spent starting tablet: real 0.043s user 0.034s sys 0.011s
I20250628 07:54:35.172748 9128 raft_consensus.cc:695] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [term 1 LEADER]: Becoming Leader. State: Replica: 7c82736b81884d95a2a4d3af10daa780, State: Running, Role: LEADER
I20250628 07:54:35.173574 9128 consensus_queue.cc:237] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82736b81884d95a2a4d3af10daa780" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 41739 } }
I20250628 07:54:35.187615 8923 catalog_manager.cc:5582] T a5dbf8dde5a74176875e2fe92537a237 P 7c82736b81884d95a2a4d3af10daa780 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c82736b81884d95a2a4d3af10daa780 (127.3.91.129). New cstate: current_term: 1 leader_uuid: "7c82736b81884d95a2a4d3af10daa780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c82736b81884d95a2a4d3af10daa780" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 41739 } health_report { overall_health: HEALTHY } } }
I20250628 07:54:35.211048 3438 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250628 07:54:35.214576 3438 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7c82736b81884d95a2a4d3af10daa780 to finish bootstrapping
I20250628 07:54:37.933442 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8982
I20250628 07:54:37.958508 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 8890
2025-06-28T07:54:37Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7096 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7096 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250628 07:54:38.012365 3438 test_util.cc:276] Using random seed: 456727471
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-28T07:54:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-28T07:54:38Z Disabled control of system clock
I20250628 07:54:38.064663 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34247
--webserver_interface=127.3.91.190
--webserver_port=0
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:34247 with env {}
W20250628 07:54:38.370622 9154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:38.371193 9154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:38.371610 9154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:38.403620 9154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:54:38.403926 9154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:38.404131 9154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:54:38.404325 9154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:54:38.439966 9154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:34247
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:38.441267 9154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:38.442939 9154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:38.459022 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:38.459035 9161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:38.459141 9163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:38.461411 9154 server_base.cc:1048] running on GCE node
I20250628 07:54:39.642763 9154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:39.645807 9154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:39.647174 9154 hybrid_clock.cc:648] HybridClock initialized: now 1751097279647151 us; error 47 us; skew 500 ppm
I20250628 07:54:39.647966 9154 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:39.657956 9154 webserver.cc:469] Webserver started at http://127.3.91.190:38121/ using document root <none> and password file <none>
I20250628 07:54:39.658928 9154 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:39.659163 9154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:39.659606 9154 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:39.664072 9154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/instance:
uuid: "e252540b39d6424dbe5697b5c0320796"
format_stamp: "Formatted at 2025-06-28 07:54:39 on dist-test-slave-wggc"
I20250628 07:54:39.665194 9154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal/instance:
uuid: "e252540b39d6424dbe5697b5c0320796"
format_stamp: "Formatted at 2025-06-28 07:54:39 on dist-test-slave-wggc"
I20250628 07:54:39.672283 9154 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250628 07:54:39.677846 9171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:39.678870 9154 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250628 07:54:39.679219 9154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
uuid: "e252540b39d6424dbe5697b5c0320796"
format_stamp: "Formatted at 2025-06-28 07:54:39 on dist-test-slave-wggc"
I20250628 07:54:39.679546 9154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:39.734606 9154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:39.736109 9154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:39.736588 9154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:39.806362 9154 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:34247
I20250628 07:54:39.806453 9222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:34247 every 8 connection(s)
I20250628 07:54:39.809120 9154 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
I20250628 07:54:39.814445 9223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:39.818805 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9154
I20250628 07:54:39.819424 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal/instance
I20250628 07:54:39.837785 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796: Bootstrap starting.
I20250628 07:54:39.844056 9223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:39.845791 9223 log.cc:826] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:39.850303 9223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796: No bootstrap required, opened a new log
I20250628 07:54:39.868108 9223 raft_consensus.cc:357] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:39.868788 9223 raft_consensus.cc:383] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:39.869041 9223 raft_consensus.cc:738] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e252540b39d6424dbe5697b5c0320796, State: Initialized, Role: FOLLOWER
I20250628 07:54:39.869737 9223 consensus_queue.cc:260] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:39.870256 9223 raft_consensus.cc:397] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:39.870512 9223 raft_consensus.cc:491] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:39.870823 9223 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:39.874812 9223 raft_consensus.cc:513] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:39.875578 9223 leader_election.cc:304] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e252540b39d6424dbe5697b5c0320796; no voters:
I20250628 07:54:39.877383 9223 leader_election.cc:290] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:54:39.878111 9228 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:39.880390 9228 raft_consensus.cc:695] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [term 1 LEADER]: Becoming Leader. State: Replica: e252540b39d6424dbe5697b5c0320796, State: Running, Role: LEADER
I20250628 07:54:39.881207 9228 consensus_queue.cc:237] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:39.882162 9223 sys_catalog.cc:564] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:54:39.894513 9230 sys_catalog.cc:455] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e252540b39d6424dbe5697b5c0320796. Latest consensus state: current_term: 1 leader_uuid: "e252540b39d6424dbe5697b5c0320796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } } }
I20250628 07:54:39.895323 9230 sys_catalog.cc:458] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:39.895398 9229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e252540b39d6424dbe5697b5c0320796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e252540b39d6424dbe5697b5c0320796" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } } }
I20250628 07:54:39.896023 9229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:39.903121 9238 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:54:39.912417 9238 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:54:39.931679 9238 catalog_manager.cc:1349] Generated new cluster ID: e861608e128244dfb1a42140b65ec618
I20250628 07:54:39.932019 9238 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:54:39.963028 9238 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:54:39.965112 9238 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:54:39.987783 9238 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e252540b39d6424dbe5697b5c0320796: Generated new TSK 0
I20250628 07:54:39.988932 9238 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250628 07:54:40.000499 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:0
--local_ip_for_outbound_sockets=127.3.91.129
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250628 07:54:40.308037 9247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:40.308557 9247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:40.309064 9247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:40.341018 9247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:40.341966 9247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:54:40.376271 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:40.377558 9247 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:40.379187 9247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:40.396904 9256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:40.399492 9247 server_base.cc:1048] running on GCE node
W20250628 07:54:40.399049 9253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:40.397418 9254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:41.566207 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:41.568964 9247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:41.570415 9247 hybrid_clock.cc:648] HybridClock initialized: now 1751097281570347 us; error 83 us; skew 500 ppm
I20250628 07:54:41.571213 9247 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:41.578404 9247 webserver.cc:469] Webserver started at http://127.3.91.129:44285/ using document root <none> and password file <none>
I20250628 07:54:41.579284 9247 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:41.579608 9247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:41.580037 9247 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:41.584573 9247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/instance:
uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
format_stamp: "Formatted at 2025-06-28 07:54:41 on dist-test-slave-wggc"
I20250628 07:54:41.585691 9247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal/instance:
uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
format_stamp: "Formatted at 2025-06-28 07:54:41 on dist-test-slave-wggc"
I20250628 07:54:41.593148 9247 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250628 07:54:41.599378 9263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:41.600476 9247 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250628 07:54:41.600785 9247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
format_stamp: "Formatted at 2025-06-28 07:54:41 on dist-test-slave-wggc"
I20250628 07:54:41.601112 9247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:41.672673 9247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:41.674240 9247 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:41.674837 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:41.677328 9247 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:41.681298 9247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:41.681541 9247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:41.681852 9247 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:41.682013 9247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:41.815075 9247 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:45525
I20250628 07:54:41.815193 9375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:45525 every 8 connection(s)
I20250628 07:54:41.818523 9247 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
I20250628 07:54:41.825407 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9247
I20250628 07:54:41.825867 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal/instance
I20250628 07:54:41.831786 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:0
--local_ip_for_outbound_sockets=127.3.91.130
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 07:54:41.840525 9376 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:54:41.841022 9376 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:41.842051 9376 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:54:41.844501 9188 ts_manager.cc:194] Registered new tserver with Master: cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:54:41.846423 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:38399
W20250628 07:54:42.132529 9380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:42.133023 9380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:42.133512 9380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:42.164952 9380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:42.165809 9380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:54:42.200244 9380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:42.201467 9380 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:42.203052 9380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:42.219168 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:42.849817 9376 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
W20250628 07:54:42.219178 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:42.219241 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:43.398041 9388 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1176 milliseconds
I20250628 07:54:43.398161 9380 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:54:43.399338 9380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:43.401535 9380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:43.402943 9380 hybrid_clock.cc:648] HybridClock initialized: now 1751097283402900 us; error 58 us; skew 500 ppm
I20250628 07:54:43.403738 9380 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:43.410490 9380 webserver.cc:469] Webserver started at http://127.3.91.130:38379/ using document root <none> and password file <none>
I20250628 07:54:43.411394 9380 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:43.411621 9380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:43.412063 9380 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:43.416671 9380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/instance:
uuid: "01ec04492fda409f8e9d8bed5a0aa036"
format_stamp: "Formatted at 2025-06-28 07:54:43 on dist-test-slave-wggc"
I20250628 07:54:43.417871 9380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal/instance:
uuid: "01ec04492fda409f8e9d8bed5a0aa036"
format_stamp: "Formatted at 2025-06-28 07:54:43 on dist-test-slave-wggc"
I20250628 07:54:43.424854 9380 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250628 07:54:43.430409 9396 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:43.431476 9380 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250628 07:54:43.431795 9380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
uuid: "01ec04492fda409f8e9d8bed5a0aa036"
format_stamp: "Formatted at 2025-06-28 07:54:43 on dist-test-slave-wggc"
I20250628 07:54:43.432125 9380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:43.484587 9380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:43.486085 9380 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:43.486522 9380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:43.489071 9380 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:43.493091 9380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:43.493307 9380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:43.493563 9380 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:43.493757 9380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250628 07:54:43.626893 9380 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:42269
I20250628 07:54:43.627005 9508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:42269 every 8 connection(s)
I20250628 07:54:43.629374 9380 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
I20250628 07:54:43.631001 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9380
I20250628 07:54:43.631417 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal/instance
I20250628 07:54:43.638882 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:0
--local_ip_for_outbound_sockets=127.3.91.131
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 07:54:43.652954 9509 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:54:43.653378 9509 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:43.654490 9509 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:54:43.656541 9188 ts_manager.cc:194] Registered new tserver with Master: 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269)
I20250628 07:54:43.658133 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:49471
W20250628 07:54:43.945786 9513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:43.946290 9513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:43.946794 9513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:43.978295 9513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:43.979205 9513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:54:44.014228 9513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=0
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:44.015542 9513 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:44.017194 9513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:44.033114 9520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:44.661202 9509 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
W20250628 07:54:44.033172 9519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:44.034051 9522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:45.211100 9521 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250628 07:54:45.211285 9513 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:54:45.215757 9513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:45.217999 9513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:45.219349 9513 hybrid_clock.cc:648] HybridClock initialized: now 1751097285219310 us; error 51 us; skew 500 ppm
I20250628 07:54:45.220142 9513 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:45.230459 9513 webserver.cc:469] Webserver started at http://127.3.91.131:33235/ using document root <none> and password file <none>
I20250628 07:54:45.231376 9513 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:45.231616 9513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:45.232108 9513 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:45.236726 9513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/instance:
uuid: "35ef99a703104e308f4484173be191a2"
format_stamp: "Formatted at 2025-06-28 07:54:45 on dist-test-slave-wggc"
I20250628 07:54:45.237871 9513 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal/instance:
uuid: "35ef99a703104e308f4484173be191a2"
format_stamp: "Formatted at 2025-06-28 07:54:45 on dist-test-slave-wggc"
I20250628 07:54:45.244946 9513 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20250628 07:54:45.250443 9529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:45.251437 9513 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250628 07:54:45.251780 9513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
uuid: "35ef99a703104e308f4484173be191a2"
format_stamp: "Formatted at 2025-06-28 07:54:45 on dist-test-slave-wggc"
I20250628 07:54:45.252126 9513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:45.306824 9513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:45.308302 9513 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:45.308754 9513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:45.311273 9513 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:45.315371 9513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250628 07:54:45.315590 9513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:45.315838 9513 ts_tablet_manager.cc:610] Registered 0 tablets
I20250628 07:54:45.316004 9513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:45.449376 9513 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:39601
I20250628 07:54:45.449479 9641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:39601 every 8 connection(s)
I20250628 07:54:45.451786 9513 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
I20250628 07:54:45.456508 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9513
I20250628 07:54:45.457059 3438 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal/instance
I20250628 07:54:45.471623 9642 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:54:45.472035 9642 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:45.472950 9642 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:54:45.475071 9188 ts_manager.cc:194] Registered new tserver with Master: 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601)
I20250628 07:54:45.476307 9188 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:41131
I20250628 07:54:45.476997 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:54:45.506043 3438 test_util.cc:276] Using random seed: 464221158
I20250628 07:54:45.545083 9188 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58060:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 07:54:45.547426 9188 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:54:45.612305 9444 tablet_service.cc:1468] Processing CreateTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 (DEFAULT_TABLE table=pre_rebuild [id=63822972f4604e999456b6ba40260b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:45.613252 9577 tablet_service.cc:1468] Processing CreateTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 (DEFAULT_TABLE table=pre_rebuild [id=63822972f4604e999456b6ba40260b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:45.614593 9444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4f53efb52a4f6fa1d79a47d47e4de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:45.614959 9577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4f53efb52a4f6fa1d79a47d47e4de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:45.614586 9311 tablet_service.cc:1468] Processing CreateTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 (DEFAULT_TABLE table=pre_rebuild [id=63822972f4604e999456b6ba40260b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:54:45.616295 9311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4f53efb52a4f6fa1d79a47d47e4de4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:45.645006 9667 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Bootstrap starting.
I20250628 07:54:45.649544 9666 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Bootstrap starting.
I20250628 07:54:45.651463 9668 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Bootstrap starting.
I20250628 07:54:45.657594 9667 tablet_bootstrap.cc:654] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:45.658816 9668 tablet_bootstrap.cc:654] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:45.659535 9666 tablet_bootstrap.cc:654] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:45.660138 9667 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:45.661204 9668 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:45.661967 9666 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:45.672317 9667 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: No bootstrap required, opened a new log
I20250628 07:54:45.672904 9667 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.006s
I20250628 07:54:45.673580 9666 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: No bootstrap required, opened a new log
I20250628 07:54:45.674167 9666 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20250628 07:54:45.674292 9668 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: No bootstrap required, opened a new log
I20250628 07:54:45.674800 9668 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.007s
I20250628 07:54:45.695124 9667 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.696063 9667 raft_consensus.cc:383] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:45.696391 9667 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ec04492fda409f8e9d8bed5a0aa036, State: Initialized, Role: FOLLOWER
I20250628 07:54:45.697374 9667 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.697952 9666 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.698632 9666 raft_consensus.cc:383] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:45.698855 9666 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ef99a703104e308f4484173be191a2, State: Initialized, Role: FOLLOWER
I20250628 07:54:45.700022 9666 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.701530 9667 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent starting tablet: real 0.028s user 0.014s sys 0.012s
I20250628 07:54:45.702257 9668 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.703241 9668 raft_consensus.cc:383] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:45.703543 9642 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
I20250628 07:54:45.703586 9668 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2eb810e5ce4cae9c0da0f9719ea127, State: Initialized, Role: FOLLOWER
I20250628 07:54:45.704794 9666 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Time spent starting tablet: real 0.030s user 0.020s sys 0.007s
I20250628 07:54:45.704562 9668 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.708786 9668 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent starting tablet: real 0.034s user 0.025s sys 0.006s
W20250628 07:54:45.709686 9643 tablet.cc:2378] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:54:45.828842 9377 tablet.cc:2378] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:54:45.863027 9672 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:54:45.863540 9672 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.865978 9672 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601), cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:54:45.879125 9331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" is_pre_election: true
I20250628 07:54:45.879382 9597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2" is_pre_election: true
I20250628 07:54:45.879845 9331 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ec04492fda409f8e9d8bed5a0aa036 in term 0.
I20250628 07:54:45.880051 9597 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01ec04492fda409f8e9d8bed5a0aa036 in term 0.
I20250628 07:54:45.880956 9397 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, cf2eb810e5ce4cae9c0da0f9719ea127; no voters:
I20250628 07:54:45.881670 9672 raft_consensus.cc:2802] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:54:45.881999 9672 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:54:45.882218 9672 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:45.886483 9672 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
W20250628 07:54:45.886758 9510 tablet.cc:2378] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:54:45.887923 9672 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 1 election: Requested vote from peers 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601), cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:54:45.888693 9597 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2"
I20250628 07:54:45.888890 9331 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
I20250628 07:54:45.889084 9597 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:45.889333 9331 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:45.894205 9597 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ec04492fda409f8e9d8bed5a0aa036 in term 1.
I20250628 07:54:45.894500 9331 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01ec04492fda409f8e9d8bed5a0aa036 in term 1.
I20250628 07:54:45.895154 9399 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, 35ef99a703104e308f4484173be191a2; no voters:
I20250628 07:54:45.895889 9672 raft_consensus.cc:2802] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:45.897400 9672 raft_consensus.cc:695] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 LEADER]: Becoming Leader. State: Replica: 01ec04492fda409f8e9d8bed5a0aa036, State: Running, Role: LEADER
I20250628 07:54:45.898237 9672 consensus_queue.cc:237] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:54:45.908757 9186 catalog_manager.cc:5582] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130). New cstate: current_term: 1 leader_uuid: "01ec04492fda409f8e9d8bed5a0aa036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } health_report { overall_health: UNKNOWN } } }
I20250628 07:54:46.106050 9331 raft_consensus.cc:1273] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Refusing update from remote peer 01ec04492fda409f8e9d8bed5a0aa036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 07:54:46.106393 9597 raft_consensus.cc:1273] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Refusing update from remote peer 01ec04492fda409f8e9d8bed5a0aa036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 07:54:46.108099 9677 consensus_queue.cc:1035] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 07:54:46.108886 9672 consensus_queue.cc:1035] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:54:46.143378 9687 mvcc.cc:204] Tried to move back new op lower bound from 7172494483871055872 to 7172494483046064128. Current Snapshot: MvccSnapshot[applied={T|T < 7172494483871055872}]
I20250628 07:54:46.144716 9685 mvcc.cc:204] Tried to move back new op lower bound from 7172494483893583872 to 7172494483046064128. Current Snapshot: MvccSnapshot[applied={T|T < 7172494483871055872}]
I20250628 07:54:46.180020 9688 mvcc.cc:204] Tried to move back new op lower bound from 7172494483871055872 to 7172494483046064128. Current Snapshot: MvccSnapshot[applied={T|T < 7172494483871055872}]
I20250628 07:54:51.342131 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9154
W20250628 07:54:51.718115 9720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:51.719003 9720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:51.753305 9720 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250628 07:54:52.164794 9638 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9560 after lost signal to thread 9514
W20250628 07:54:52.166849 9638 debug-util.cc:398] Leaking SignalData structure 0x7b08000cce80 after lost signal to thread 9641
W20250628 07:54:52.237527 9642 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:34247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:34247: connect: Connection refused (error 111)
W20250628 07:54:52.249372 9509 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:34247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:34247: connect: Connection refused (error 111)
W20250628 07:54:52.308951 9376 heartbeater.cc:646] Failed to heartbeat to 127.3.91.190:34247 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.91.190:34247: connect: Connection refused (error 111)
W20250628 07:54:53.205139 9725 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 9720
W20250628 07:54:53.206363 9725 kernel_stack_watchdog.cc:198] Thread 9720 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 07:54:53.603725 9720 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.803s user 0.632s sys 1.125s
W20250628 07:54:53.761320 9720 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.961s user 0.633s sys 1.139s
I20250628 07:54:53.879763 9720 minidump.cc:252] Setting minidump size limit to 20M
I20250628 07:54:53.882256 9720 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:53.883755 9720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:53.897235 9753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:53.898428 9754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:53.901280 9756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:53.901422 9720 server_base.cc:1048] running on GCE node
I20250628 07:54:53.902851 9720 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250628 07:54:53.903431 9720 hybrid_clock.cc:648] HybridClock initialized: now 1751097293903402 us; error 163016 us; skew 500 ppm
I20250628 07:54:53.904420 9720 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:53.910053 9720 webserver.cc:469] Webserver started at http://0.0.0.0:46745/ using document root <none> and password file <none>
I20250628 07:54:53.911214 9720 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:53.911513 9720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:53.912072 9720 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250628 07:54:53.919016 9720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/instance:
uuid: "8985e35ad2e54225b68a9fa8b9ca7351"
format_stamp: "Formatted at 2025-06-28 07:54:53 on dist-test-slave-wggc"
I20250628 07:54:53.920612 9720 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal/instance:
uuid: "8985e35ad2e54225b68a9fa8b9ca7351"
format_stamp: "Formatted at 2025-06-28 07:54:53 on dist-test-slave-wggc"
I20250628 07:54:53.928766 9720 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250628 07:54:53.935536 9761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:53.936622 9720 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250628 07:54:53.936944 9720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
uuid: "8985e35ad2e54225b68a9fa8b9ca7351"
format_stamp: "Formatted at 2025-06-28 07:54:53 on dist-test-slave-wggc"
I20250628 07:54:53.937258 9720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:54.050293 9720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:54.051705 9720 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:54.052143 9720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:54.056805 9720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:54:54.070910 9720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Bootstrap starting.
I20250628 07:54:54.078260 9720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Neither blocks nor log segments found. Creating new log.
I20250628 07:54:54.080471 9720 log.cc:826] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:54.088778 9720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: No bootstrap required, opened a new log
I20250628 07:54:54.111060 9720 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER }
I20250628 07:54:54.111577 9720 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:54:54.111794 9720 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8985e35ad2e54225b68a9fa8b9ca7351, State: Initialized, Role: FOLLOWER
I20250628 07:54:54.112452 9720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER }
I20250628 07:54:54.112967 9720 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:54.113205 9720 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:54.113521 9720 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:54:54.117449 9720 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER }
I20250628 07:54:54.118248 9720 leader_election.cc:304] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8985e35ad2e54225b68a9fa8b9ca7351; no voters:
I20250628 07:54:54.120481 9720 leader_election.cc:290] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250628 07:54:54.120745 9769 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:54:54.125711 9769 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 LEADER]: Becoming Leader. State: Replica: 8985e35ad2e54225b68a9fa8b9ca7351, State: Running, Role: LEADER
I20250628 07:54:54.126746 9769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER }
I20250628 07:54:54.133957 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER } }
I20250628 07:54:54.134991 9771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8985e35ad2e54225b68a9fa8b9ca7351. Latest consensus state: current_term: 1 leader_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER } }
I20250628 07:54:54.135495 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:54.135468 9771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:54.149679 9720 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: stopping tablet replica
I20250628 07:54:54.150251 9720 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 LEADER]: Raft consensus shutting down.
I20250628 07:54:54.150673 9720 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250628 07:54:54.152962 9720 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250628 07:54:54.153379 9720 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250628 07:54:54.217564 9720 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250628 07:54:54.408156 9725 debug-util.cc:398] Leaking SignalData structure 0x7b0800036060 after lost signal to thread 9720
I20250628 07:54:55.247808 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9247
I20250628 07:54:55.284619 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9380
I20250628 07:54:55.330843 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9513
I20250628 07:54:55.368101 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34247
--webserver_interface=127.3.91.190
--webserver_port=38121
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.91.190:34247 with env {}
W20250628 07:54:55.669108 9780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:55.669688 9780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:55.670109 9780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:55.700834 9780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250628 07:54:55.701133 9780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:55.701341 9780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250628 07:54:55.701535 9780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250628 07:54:55.736160 9780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.91.190:34247
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.91.190:34247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.91.190
--webserver_port=38121
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:55.737413 9780 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:55.739005 9780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:55.754949 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:55.755424 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:55.755017 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:54:55.757265 9780 server_base.cc:1048] running on GCE node
I20250628 07:54:56.945222 9780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:56.947819 9780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:56.949173 9780 hybrid_clock.cc:648] HybridClock initialized: now 1751097296949132 us; error 57 us; skew 500 ppm
I20250628 07:54:56.950001 9780 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:56.956997 9780 webserver.cc:469] Webserver started at http://127.3.91.190:38121/ using document root <none> and password file <none>
I20250628 07:54:56.957979 9780 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:56.958185 9780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:56.965984 9780 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250628 07:54:56.970458 9796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:56.971514 9780 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250628 07:54:56.971812 9780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
uuid: "8985e35ad2e54225b68a9fa8b9ca7351"
format_stamp: "Formatted at 2025-06-28 07:54:53 on dist-test-slave-wggc"
I20250628 07:54:56.973748 9780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:57.033493 9780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:57.034970 9780 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:57.035387 9780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:57.105597 9780 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.190:34247
I20250628 07:54:57.105722 9847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.190:34247 every 8 connection(s)
I20250628 07:54:57.108561 9780 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb
I20250628 07:54:57.114925 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9780
I20250628 07:54:57.116969 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.129:45525
--local_ip_for_outbound_sockets=127.3.91.129
--tserver_master_addrs=127.3.91.190:34247
--webserver_port=44285
--webserver_interface=127.3.91.129
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 07:54:57.121026 9848 sys_catalog.cc:263] Verifying existing consensus state
I20250628 07:54:57.136526 9848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Bootstrap starting.
I20250628 07:54:57.146500 9848 log.cc:826] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Log is configured to *not* fsync() on all Append() calls
I20250628 07:54:57.158955 9848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:54:57.159859 9848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Bootstrap complete.
I20250628 07:54:57.180328 9848 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:57.181030 9848 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8985e35ad2e54225b68a9fa8b9ca7351, State: Initialized, Role: FOLLOWER
I20250628 07:54:57.181808 9848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:57.182294 9848 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250628 07:54:57.182559 9848 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250628 07:54:57.182855 9848 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:54:57.186910 9848 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:57.187551 9848 leader_election.cc:304] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8985e35ad2e54225b68a9fa8b9ca7351; no voters:
I20250628 07:54:57.189375 9848 leader_election.cc:290] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250628 07:54:57.189800 9852 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:54:57.192912 9852 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [term 2 LEADER]: Becoming Leader. State: Replica: 8985e35ad2e54225b68a9fa8b9ca7351, State: Running, Role: LEADER
I20250628 07:54:57.194064 9848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: configured and running, proceeding with master startup.
I20250628 07:54:57.193856 9852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } }
I20250628 07:54:57.205261 9853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } } }
I20250628 07:54:57.206924 9853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:57.207464 9854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8985e35ad2e54225b68a9fa8b9ca7351. Latest consensus state: current_term: 2 leader_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8985e35ad2e54225b68a9fa8b9ca7351" member_type: VOTER last_known_addr { host: "127.3.91.190" port: 34247 } } }
I20250628 07:54:57.210978 9854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351 [sys.catalog]: This master's current role is: LEADER
I20250628 07:54:57.216835 9858 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250628 07:54:57.231010 9858 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=bbf45f30f6c344a9888f6bfae80b2d42]
I20250628 07:54:57.239750 9858 tablet_loader.cc:96] loaded metadata for tablet af4f53efb52a4f6fa1d79a47d47e4de4 (table pre_rebuild [id=bbf45f30f6c344a9888f6bfae80b2d42])
I20250628 07:54:57.241657 9858 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250628 07:54:57.274627 9858 catalog_manager.cc:1349] Generated new cluster ID: 72136ec9359a49bda79f50972ef29c31
I20250628 07:54:57.274945 9858 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250628 07:54:57.302002 9858 catalog_manager.cc:1372] Generated new certificate authority record
I20250628 07:54:57.304111 9858 catalog_manager.cc:1506] Loading token signing keys...
I20250628 07:54:57.327247 9858 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Generated new TSK 0
I20250628 07:54:57.328495 9858 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250628 07:54:57.476701 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:57.477216 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:57.477767 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:57.509073 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:57.509984 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.129
I20250628 07:54:57.544689 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.129:45525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.91.129
--webserver_port=44285
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.129
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:57.546013 9850 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:57.547602 9850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:57.564452 9876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:57.566021 9879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:57.564764 9877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:54:59.031852 9878 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1466 milliseconds
I20250628 07:54:59.031986 9850 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:54:59.033277 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:54:59.036134 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:54:59.037690 9850 hybrid_clock.cc:648] HybridClock initialized: now 1751097299037631 us; error 59 us; skew 500 ppm
I20250628 07:54:59.038523 9850 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:54:59.050089 9850 webserver.cc:469] Webserver started at http://127.3.91.129:44285/ using document root <none> and password file <none>
I20250628 07:54:59.051019 9850 fs_manager.cc:362] Metadata directory not provided
I20250628 07:54:59.051234 9850 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:54:59.059237 9850 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250628 07:54:59.064165 9886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:54:59.065204 9850 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250628 07:54:59.065508 9850 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
format_stamp: "Formatted at 2025-06-28 07:54:41 on dist-test-slave-wggc"
I20250628 07:54:59.067440 9850 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:54:59.119237 9850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:54:59.120723 9850 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:54:59.121162 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:54:59.124209 9850 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:54:59.130491 9893 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:54:59.138255 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:54:59.138523 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.002s
I20250628 07:54:59.138799 9850 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:54:59.144356 9850 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:54:59.144711 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.002s
I20250628 07:54:59.145030 9893 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Bootstrap starting.
I20250628 07:54:59.362154 9850 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.129:45525
I20250628 07:54:59.362331 10000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.129:45525 every 8 connection(s)
I20250628 07:54:59.365211 9850 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb
I20250628 07:54:59.374079 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 9850
I20250628 07:54:59.376932 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.130:42269
--local_ip_for_outbound_sockets=127.3.91.130
--tserver_master_addrs=127.3.91.190:34247
--webserver_port=38379
--webserver_interface=127.3.91.130
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 07:54:59.415462 10001 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:54:59.416013 10001 heartbeater.cc:461] Registering TS with master...
I20250628 07:54:59.417296 10001 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:54:59.422297 9813 ts_manager.cc:194] Registered new tserver with Master: cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:54:59.430354 9813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.129:46459
I20250628 07:54:59.464323 9893 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Log is configured to *not* fsync() on all Append() calls
W20250628 07:54:59.833222 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:54:59.833772 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:54:59.834287 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:54:59.865159 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:54:59.866539 10005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.130
I20250628 07:54:59.901818 10005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.130:42269
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.91.130
--webserver_port=38379
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.130
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:54:59.903113 10005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:54:59.905079 10005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:54:59.921823 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:55:00.435796 10001 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
W20250628 07:55:01.749110 10005 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.831s user 0.612s sys 1.120s
W20250628 07:54:59.922324 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:55:01.750296 10005 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.833s user 0.612s sys 1.120s
W20250628 07:55:01.318302 10011 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 10005
W20250628 07:55:01.751699 10014 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1829 milliseconds
W20250628 07:55:01.752878 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:55:01.752830 10005 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250628 07:55:01.756336 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:55:01.759043 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:55:01.760555 10005 hybrid_clock.cc:648] HybridClock initialized: now 1751097301760502 us; error 66 us; skew 500 ppm
I20250628 07:55:01.761402 10005 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:55:01.768395 10005 webserver.cc:469] Webserver started at http://127.3.91.130:38379/ using document root <none> and password file <none>
I20250628 07:55:01.769315 10005 fs_manager.cc:362] Metadata directory not provided
I20250628 07:55:01.769531 10005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:55:01.777926 10005 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20250628 07:55:01.783074 10022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:55:01.784225 10005 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250628 07:55:01.784543 10005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
uuid: "01ec04492fda409f8e9d8bed5a0aa036"
format_stamp: "Formatted at 2025-06-28 07:54:43 on dist-test-slave-wggc"
I20250628 07:55:01.786561 10005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:55:01.858830 10005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:55:01.860373 10005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:55:01.860831 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:55:01.863948 10005 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:55:01.870558 10029 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:55:01.878149 10005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:55:01.878374 10005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.003s
I20250628 07:55:01.878700 10005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:55:01.883392 10005 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:55:01.883674 10005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250628 07:55:01.884084 10029 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Bootstrap starting.
I20250628 07:55:02.059012 10005 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.130:42269
I20250628 07:55:02.059192 10135 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.130:42269 every 8 connection(s)
I20250628 07:55:02.062861 10005 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb
I20250628 07:55:02.071523 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 10005
I20250628 07:55:02.073405 3438 external_mini_cluster.cc:1351] Running /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
/tmp/dist-test-task7GuWg7/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.91.131:39601
--local_ip_for_outbound_sockets=127.3.91.131
--tserver_master_addrs=127.3.91.190:34247
--webserver_port=33235
--webserver_interface=127.3.91.131
--builtin_ntp_servers=127.3.91.148:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250628 07:55:02.121682 10136 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:55:02.122244 10136 heartbeater.cc:461] Registering TS with master...
I20250628 07:55:02.123442 10136 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:55:02.127364 9813 ts_manager.cc:194] Registered new tserver with Master: 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269)
I20250628 07:55:02.130347 9813 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.130:55105
I20250628 07:55:02.214413 10029 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Log is configured to *not* fsync() on all Append() calls
W20250628 07:55:02.541328 10140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250628 07:55:02.542006 10140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250628 07:55:02.542764 10140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250628 07:55:02.597882 10140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250628 07:55:02.599248 10140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.91.131
I20250628 07:55:02.659336 10140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.91.148:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.91.131:39601
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.91.131
--webserver_port=33235
--tserver_master_addrs=127.3.91.190:34247
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.91.131
--log_dir=/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f0776864a65c19ebdd909053f3e624e8bcc25cee
build type FASTDEBUG
built by None at 28 Jun 2025 07:43:20 UTC on 5fd53c4cbb9d
build id 6809
TSAN enabled
I20250628 07:55:02.660918 10140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250628 07:55:02.663057 10140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250628 07:55:02.682821 10147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:55:03.133903 10136 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
I20250628 07:55:03.395965 9893 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:55:03.397310 9893 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Bootstrap complete.
I20250628 07:55:03.399490 9893 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent bootstrapping tablet: real 4.255s user 3.858s sys 0.091s
I20250628 07:55:03.421972 9893 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:03.426072 9893 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2eb810e5ce4cae9c0da0f9719ea127, State: Initialized, Role: FOLLOWER
I20250628 07:55:03.427372 9893 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:03.437458 9893 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent starting tablet: real 0.038s user 0.031s sys 0.003s
W20250628 07:55:02.683012 10148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250628 07:55:04.081234 10146 debug-util.cc:398] Leaking SignalData structure 0x7b0800000a80 after lost signal to thread 10140
W20250628 07:55:04.180927 10146 kernel_stack_watchdog.cc:198] Thread 10140 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 395ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250628 07:55:04.186693 10140 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.503s user 0.000s sys 0.002s
W20250628 07:55:04.187096 10140 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.504s user 0.000s sys 0.002s
W20250628 07:55:04.188822 10150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250628 07:55:04.188758 10140 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250628 07:55:04.189275 10149 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1503 milliseconds
I20250628 07:55:04.190403 10140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250628 07:55:04.192806 10140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250628 07:55:04.194275 10140 hybrid_clock.cc:648] HybridClock initialized: now 1751097304194214 us; error 63 us; skew 500 ppm
I20250628 07:55:04.195079 10140 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250628 07:55:04.208565 10140 webserver.cc:469] Webserver started at http://127.3.91.131:33235/ using document root <none> and password file <none>
I20250628 07:55:04.209488 10140 fs_manager.cc:362] Metadata directory not provided
I20250628 07:55:04.209748 10140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250628 07:55:04.217728 10140 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250628 07:55:04.222797 10158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250628 07:55:04.223894 10140 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250628 07:55:04.224197 10140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data,/tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
uuid: "35ef99a703104e308f4484173be191a2"
format_stamp: "Formatted at 2025-06-28 07:54:45 on dist-test-slave-wggc"
I20250628 07:55:04.226205 10140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250628 07:55:04.297587 10140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250628 07:55:04.299104 10140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250628 07:55:04.299543 10140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250628 07:55:04.302851 10140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250628 07:55:04.309538 10165 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250628 07:55:04.316922 10140 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250628 07:55:04.317139 10140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250628 07:55:04.317368 10140 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250628 07:55:04.321975 10140 ts_tablet_manager.cc:610] Registered 1 tablets
I20250628 07:55:04.322225 10140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.005s
I20250628 07:55:04.322626 10165 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Bootstrap starting.
I20250628 07:55:04.535619 10140 rpc_server.cc:307] RPC server started. Bound to: 127.3.91.131:39601
I20250628 07:55:04.535938 10271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.3.91.131:39601 every 8 connection(s)
I20250628 07:55:04.539286 10140 server_base.cc:1180] Dumped server information to /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb
I20250628 07:55:04.543681 3438 external_mini_cluster.cc:1413] Started /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu as pid 10140
I20250628 07:55:04.593312 10272 heartbeater.cc:344] Connected to a master server at 127.3.91.190:34247
I20250628 07:55:04.593865 10272 heartbeater.cc:461] Registering TS with master...
I20250628 07:55:04.595113 10272 heartbeater.cc:507] Master 127.3.91.190:34247 requested a full tablet report, sending...
I20250628 07:55:04.599045 9812 ts_manager.cc:194] Registered new tserver with Master: 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601)
I20250628 07:55:04.602273 9812 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.91.131:37493
I20250628 07:55:04.603740 3438 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250628 07:55:04.674546 10165 log.cc:826] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Log is configured to *not* fsync() on all Append() calls
I20250628 07:55:04.734480 10284 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:55:04.735086 10284 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:04.738132 10284 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269), 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601)
I20250628 07:55:04.782490 10227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2" is_pre_election: true
W20250628 07:55:04.796000 9889 leader_election.cc:343] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601): Illegal state: must be running to vote when last-logged opid is not known
I20250628 07:55:04.781791 10091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "01ec04492fda409f8e9d8bed5a0aa036" is_pre_election: true
W20250628 07:55:04.798900 9888 leader_election.cc:343] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269): Illegal state: must be running to vote when last-logged opid is not known
I20250628 07:55:04.799321 9888 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf2eb810e5ce4cae9c0da0f9719ea127; no voters: 01ec04492fda409f8e9d8bed5a0aa036, 35ef99a703104e308f4484173be191a2
I20250628 07:55:04.800560 10284 raft_consensus.cc:2747] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 07:55:05.200100 10029 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:55:05.200933 10029 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Bootstrap complete.
I20250628 07:55:05.202297 10029 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent bootstrapping tablet: real 3.319s user 3.040s sys 0.069s
I20250628 07:55:05.207778 10029 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:05.209771 10029 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ec04492fda409f8e9d8bed5a0aa036, State: Initialized, Role: FOLLOWER
I20250628 07:55:05.210403 10029 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:05.213397 10029 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20250628 07:55:05.606667 10272 heartbeater.cc:499] Master 127.3.91.190:34247 was elected leader, sending a full tablet report...
W20250628 07:55:05.636109 9844 debug-util.cc:398] Leaking SignalData structure 0x7b080006f3a0 after lost signal to thread 9781
W20250628 07:55:05.636786 9844 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8c20 after lost signal to thread 9847
I20250628 07:55:06.637389 10165 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250628 07:55:06.638208 10165 tablet_bootstrap.cc:492] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Bootstrap complete.
I20250628 07:55:06.639513 10165 ts_tablet_manager.cc:1397] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Time spent bootstrapping tablet: real 2.317s user 2.223s sys 0.072s
I20250628 07:55:06.644665 10165 raft_consensus.cc:357] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.646696 10165 raft_consensus.cc:738] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ef99a703104e308f4484173be191a2, State: Initialized, Role: FOLLOWER
I20250628 07:55:06.647398 10165 consensus_queue.cc:260] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.650162 10165 ts_tablet_manager.cc:1428] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20250628 07:55:06.679723 10292 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:55:06.680186 10292 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.681452 10293 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:55:06.681921 10293 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.683219 10292 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601), cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:55:06.683755 10293 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269), 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601)
I20250628 07:55:06.686300 10091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "01ec04492fda409f8e9d8bed5a0aa036" is_pre_election: true
I20250628 07:55:06.686880 10091 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf2eb810e5ce4cae9c0da0f9719ea127 in term 1.
I20250628 07:55:06.686945 10227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2" is_pre_election: true
I20250628 07:55:06.687606 10227 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf2eb810e5ce4cae9c0da0f9719ea127 in term 1.
I20250628 07:55:06.688740 9888 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, cf2eb810e5ce4cae9c0da0f9719ea127; no voters:
I20250628 07:55:06.702790 10298 raft_consensus.cc:2802] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250628 07:55:06.703269 10298 raft_consensus.cc:491] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:55:06.703655 10298 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:55:06.716099 10298 raft_consensus.cc:513] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.719058 10091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "01ec04492fda409f8e9d8bed5a0aa036"
I20250628 07:55:06.719599 10091 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:55:06.719712 10227 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2"
I20250628 07:55:06.720270 10227 raft_consensus.cc:3058] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Advancing to term 2
I20250628 07:55:06.722883 10226 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "35ef99a703104e308f4484173be191a2" is_pre_election: true
I20250628 07:55:06.723124 9956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af4f53efb52a4f6fa1d79a47d47e4de4" candidate_uuid: "01ec04492fda409f8e9d8bed5a0aa036" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" is_pre_election: true
I20250628 07:55:06.724138 9956 raft_consensus.cc:2391] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 01ec04492fda409f8e9d8bed5a0aa036 in current term 2: Already voted for candidate cf2eb810e5ce4cae9c0da0f9719ea127 in this term.
I20250628 07:55:06.725420 10298 leader_election.cc:290] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 election: Requested vote from peers 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269), 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601)
I20250628 07:55:06.725659 10023 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036; no voters: 35ef99a703104e308f4484173be191a2, cf2eb810e5ce4cae9c0da0f9719ea127
I20250628 07:55:06.728255 10091 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2eb810e5ce4cae9c0da0f9719ea127 in term 2.
I20250628 07:55:06.728629 10292 raft_consensus.cc:2747] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250628 07:55:06.730490 9888 leader_election.cc:304] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, cf2eb810e5ce4cae9c0da0f9719ea127; no voters:
I20250628 07:55:06.731263 10298 raft_consensus.cc:2802] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 FOLLOWER]: Leader election won for term 2
I20250628 07:55:06.731773 10298 raft_consensus.cc:695] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 LEADER]: Becoming Leader. State: Replica: cf2eb810e5ce4cae9c0da0f9719ea127, State: Running, Role: LEADER
I20250628 07:55:06.731815 10227 raft_consensus.cc:2466] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf2eb810e5ce4cae9c0da0f9719ea127 in term 2.
I20250628 07:55:06.733089 10298 consensus_queue.cc:237] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:06.742659 9812 catalog_manager.cc:5582] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 reported cstate change: term changed from 0 to 2, leader changed from <none> to cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129), VOTER 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130) added, VOTER 35ef99a703104e308f4484173be191a2 (127.3.91.131) added, VOTER cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129) added. New cstate: current_term: 2 leader_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } health_report { overall_health: HEALTHY } } }
I20250628 07:55:07.158771 10091 raft_consensus.cc:1273] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Refusing update from remote peer cf2eb810e5ce4cae9c0da0f9719ea127: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250628 07:55:07.160741 10298 consensus_queue.cc:1035] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250628 07:55:07.190178 10227 raft_consensus.cc:1273] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 2 FOLLOWER]: Refusing update from remote peer cf2eb810e5ce4cae9c0da0f9719ea127: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250628 07:55:07.192296 10293 consensus_queue.cc:1035] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250628 07:55:07.231662 3438 scanner-internal.cc:458] Time spent opening tablet: real 2.552s user 0.008s sys 0.001s
I20250628 07:55:07.250628 9956 consensus_queue.cc:237] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:07.260264 10090 raft_consensus.cc:1273] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Refusing update from remote peer cf2eb810e5ce4cae9c0da0f9719ea127: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250628 07:55:07.261781 10298 consensus_queue.cc:1035] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250628 07:55:07.268883 10298 raft_consensus.cc:2953] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 35ef99a703104e308f4484173be191a2 (127.3.91.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } } }
I20250628 07:55:07.271472 10091 raft_consensus.cc:2953] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 35ef99a703104e308f4484173be191a2 (127.3.91.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } } }
I20250628 07:55:07.290526 9812 catalog_manager.cc:5582] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 reported cstate change: config changed from index -1 to 207, VOTER 35ef99a703104e308f4484173be191a2 (127.3.91.131) evicted. New cstate: current_term: 2 leader_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } } }
I20250628 07:55:07.294447 9797 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet af4f53efb52a4f6fa1d79a47d47e4de4 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 07:55:07.334053 9956 consensus_queue.cc:237] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:07.338326 10298 raft_consensus.cc:2953] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } } }
I20250628 07:55:07.366393 9797 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet af4f53efb52a4f6fa1d79a47d47e4de4 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250628 07:55:07.371136 9813 catalog_manager.cc:5582] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 reported cstate change: config changed from index 207 to 208, VOTER 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130) evicted. New cstate: current_term: 2 leader_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } health_report { overall_health: HEALTHY } } }
I20250628 07:55:07.411693 10207 tablet_service.cc:1515] Processing DeleteTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 with delete_type TABLET_DATA_TOMBSTONED (TS 35ef99a703104e308f4484173be191a2 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:36136
I20250628 07:55:07.423064 10071 tablet_service.cc:1515] Processing DeleteTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 with delete_type TABLET_DATA_TOMBSTONED (TS 01ec04492fda409f8e9d8bed5a0aa036 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:57408
I20250628 07:55:07.427762 10320 tablet_replica.cc:331] stopping tablet replica
I20250628 07:55:07.438081 10321 tablet_replica.cc:331] stopping tablet replica
I20250628 07:55:07.438301 10320 raft_consensus.cc:2241] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 07:55:07.439002 10320 raft_consensus.cc:2270] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 07:55:07.441022 10321 raft_consensus.cc:2241] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250628 07:55:07.446887 10321 raft_consensus.cc:2270] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 07:55:07.486222 10320 ts_tablet_manager.cc:1905] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 07:55:07.503845 10321 ts_tablet_manager.cc:1905] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250628 07:55:07.509578 10320 ts_tablet_manager.cc:1918] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250628 07:55:07.510118 10320 log.cc:1199] T af4f53efb52a4f6fa1d79a47d47e4de4 P 35ef99a703104e308f4484173be191a2: Deleting WAL directory at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/wal/wals/af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:07.512415 9799 catalog_manager.cc:4928] TS 35ef99a703104e308f4484173be191a2 (127.3.91.131:39601): tablet af4f53efb52a4f6fa1d79a47d47e4de4 (table pre_rebuild [id=bbf45f30f6c344a9888f6bfae80b2d42]) successfully deleted
I20250628 07:55:07.540390 10321 ts_tablet_manager.cc:1918] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250628 07:55:07.540983 10321 log.cc:1199] T af4f53efb52a4f6fa1d79a47d47e4de4 P 01ec04492fda409f8e9d8bed5a0aa036: Deleting WAL directory at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/wal/wals/af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:07.543054 9798 catalog_manager.cc:4928] TS 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269): tablet af4f53efb52a4f6fa1d79a47d47e4de4 (table pre_rebuild [id=bbf45f30f6c344a9888f6bfae80b2d42]) successfully deleted
I20250628 07:55:08.129172 10071 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:55:08.146643 10207 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:55:08.157359 9936 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
8985e35ad2e54225b68a9fa8b9ca7351 | 127.3.91.190:34247 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:44751 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
01ec04492fda409f8e9d8bed5a0aa036 | 127.3.91.130:42269 | HEALTHY | <none> | 0 | 0
35ef99a703104e308f4484173be191a2 | 127.3.91.131:39601 | HEALTHY | <none> | 0 | 0
cf2eb810e5ce4cae9c0da0f9719ea127 | 127.3.91.129:45525 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.91.129 | experimental | 127.3.91.129:45525
local_ip_for_outbound_sockets | 127.3.91.130 | experimental | 127.3.91.130:42269
local_ip_for_outbound_sockets | 127.3.91.131 | experimental | 127.3.91.131:39601
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb | hidden | 127.3.91.129:45525
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.91.130:42269
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.91.131:39601
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:44751 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 07:55:08.449087 3438 log_verifier.cc:126] Checking tablet af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:08.718572 3438 log_verifier.cc:177] Verified matching terms for 208 ops in tablet af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:08.720956 9812 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59736:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 07:55:08.721417 9812 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59736:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250628 07:55:08.734001 9812 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 8985e35ad2e54225b68a9fa8b9ca7351: Sending DeleteTablet for 1 replicas of tablet af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:08.736143 3438 test_util.cc:276] Using random seed: 487451250
I20250628 07:55:08.736027 9936 tablet_service.cc:1515] Processing DeleteTablet for tablet af4f53efb52a4f6fa1d79a47d47e4de4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-28 07:55:08 UTC) from {username='slave'} at 127.0.0.1:43446
I20250628 07:55:08.738073 10354 tablet_replica.cc:331] stopping tablet replica
I20250628 07:55:08.739001 10354 raft_consensus.cc:2241] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 LEADER]: Raft consensus shutting down.
I20250628 07:55:08.739928 10354 raft_consensus.cc:2270] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250628 07:55:08.772917 9812 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54540:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250628 07:55:08.775404 9812 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250628 07:55:08.797670 10207 tablet_service.cc:1468] Processing CreateTablet for tablet 942734a979c44623bcb75e1cf80d6f87 (DEFAULT_TABLE table=post_rebuild [id=3e9a5f46759e4a9e9d4b73aaa377e50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:55:08.799121 10207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942734a979c44623bcb75e1cf80d6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:55:08.799919 10354 ts_tablet_manager.cc:1905] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250628 07:55:08.809855 10071 tablet_service.cc:1468] Processing CreateTablet for tablet 942734a979c44623bcb75e1cf80d6f87 (DEFAULT_TABLE table=post_rebuild [id=3e9a5f46759e4a9e9d4b73aaa377e50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:55:08.811167 10071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942734a979c44623bcb75e1cf80d6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:55:08.815671 9936 tablet_service.cc:1468] Processing CreateTablet for tablet 942734a979c44623bcb75e1cf80d6f87 (DEFAULT_TABLE table=post_rebuild [id=3e9a5f46759e4a9e9d4b73aaa377e50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250628 07:55:08.817094 9936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942734a979c44623bcb75e1cf80d6f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250628 07:55:08.828459 10361 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: Bootstrap starting.
I20250628 07:55:08.839778 10361 tablet_bootstrap.cc:654] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: Neither blocks nor log segments found. Creating new log.
I20250628 07:55:08.846374 10363 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: Bootstrap starting.
I20250628 07:55:08.848757 10354 ts_tablet_manager.cc:1918] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250628 07:55:08.849261 10354 log.cc:1199] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Deleting WAL directory at /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/wal/wals/af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:08.850327 10354 ts_tablet_manager.cc:1939] T af4f53efb52a4f6fa1d79a47d47e4de4 P cf2eb810e5ce4cae9c0da0f9719ea127: Deleting consensus metadata
I20250628 07:55:08.856509 10361 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: No bootstrap required, opened a new log
I20250628 07:55:08.857007 10361 ts_tablet_manager.cc:1397] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.000s
I20250628 07:55:08.858078 9797 catalog_manager.cc:4928] TS cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525): tablet af4f53efb52a4f6fa1d79a47d47e4de4 (table pre_rebuild [id=bbf45f30f6c344a9888f6bfae80b2d42]) successfully deleted
I20250628 07:55:08.861299 10364 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: Bootstrap starting.
I20250628 07:55:08.859858 10361 raft_consensus.cc:357] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.862332 10361 raft_consensus.cc:383] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:55:08.863564 10363 tablet_bootstrap.cc:654] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: Neither blocks nor log segments found. Creating new log.
I20250628 07:55:08.867646 10361 raft_consensus.cc:738] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ef99a703104e308f4484173be191a2, State: Initialized, Role: FOLLOWER
I20250628 07:55:08.868559 10361 consensus_queue.cc:260] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.872098 10364 tablet_bootstrap.cc:654] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: Neither blocks nor log segments found. Creating new log.
I20250628 07:55:08.879432 10361 ts_tablet_manager.cc:1428] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: Time spent starting tablet: real 0.022s user 0.005s sys 0.004s
I20250628 07:55:08.884207 10363 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: No bootstrap required, opened a new log
I20250628 07:55:08.884718 10363 ts_tablet_manager.cc:1397] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.013s
I20250628 07:55:08.887408 10363 raft_consensus.cc:357] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.888130 10363 raft_consensus.cc:383] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:55:08.888434 10363 raft_consensus.cc:738] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ec04492fda409f8e9d8bed5a0aa036, State: Initialized, Role: FOLLOWER
I20250628 07:55:08.889346 10363 consensus_queue.cc:260] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.893426 10366 raft_consensus.cc:491] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250628 07:55:08.895538 10366 raft_consensus.cc:513] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.902604 10364 tablet_bootstrap.cc:492] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: No bootstrap required, opened a new log
I20250628 07:55:08.903067 10364 ts_tablet_manager.cc:1397] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.000s
I20250628 07:55:08.904434 10363 ts_tablet_manager.cc:1428] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: Time spent starting tablet: real 0.019s user 0.006s sys 0.010s
I20250628 07:55:08.917331 10091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942734a979c44623bcb75e1cf80d6f87" candidate_uuid: "35ef99a703104e308f4484173be191a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ec04492fda409f8e9d8bed5a0aa036" is_pre_election: true
I20250628 07:55:08.917946 10091 raft_consensus.cc:2466] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ef99a703104e308f4484173be191a2 in term 0.
I20250628 07:55:08.919421 10160 leader_election.cc:304] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, 35ef99a703104e308f4484173be191a2; no voters:
I20250628 07:55:08.926034 10366 leader_election.cc:290] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269), cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:55:08.927114 10366 raft_consensus.cc:2802] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250628 07:55:08.928900 10364 raft_consensus.cc:357] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.929919 10366 raft_consensus.cc:491] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250628 07:55:08.930686 10366 raft_consensus.cc:3058] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:55:08.933235 10364 raft_consensus.cc:383] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250628 07:55:08.933651 10364 raft_consensus.cc:738] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2eb810e5ce4cae9c0da0f9719ea127, State: Initialized, Role: FOLLOWER
I20250628 07:55:08.934561 10364 consensus_queue.cc:260] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.944355 10364 ts_tablet_manager.cc:1428] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: Time spent starting tablet: real 0.041s user 0.005s sys 0.004s
I20250628 07:55:08.942893 10366 raft_consensus.cc:513] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.947278 10091 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942734a979c44623bcb75e1cf80d6f87" candidate_uuid: "35ef99a703104e308f4484173be191a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ec04492fda409f8e9d8bed5a0aa036"
I20250628 07:55:08.947805 10091 raft_consensus.cc:3058] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:55:08.950004 10366 leader_election.cc:290] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [CANDIDATE]: Term 1 election: Requested vote from peers 01ec04492fda409f8e9d8bed5a0aa036 (127.3.91.130:42269), cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525)
I20250628 07:55:08.949823 9956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942734a979c44623bcb75e1cf80d6f87" candidate_uuid: "35ef99a703104e308f4484173be191a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" is_pre_election: true
I20250628 07:55:08.950488 9956 raft_consensus.cc:2466] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ef99a703104e308f4484173be191a2 in term 0.
I20250628 07:55:08.951612 9956 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "942734a979c44623bcb75e1cf80d6f87" candidate_uuid: "35ef99a703104e308f4484173be191a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127"
I20250628 07:55:08.952131 9956 raft_consensus.cc:3058] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 0 FOLLOWER]: Advancing to term 1
I20250628 07:55:08.954208 10091 raft_consensus.cc:2466] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ef99a703104e308f4484173be191a2 in term 1.
I20250628 07:55:08.955446 10160 leader_election.cc:304] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ec04492fda409f8e9d8bed5a0aa036, 35ef99a703104e308f4484173be191a2; no voters:
I20250628 07:55:08.956071 10366 raft_consensus.cc:2802] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 1 FOLLOWER]: Leader election won for term 1
I20250628 07:55:08.956655 10366 raft_consensus.cc:695] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [term 1 LEADER]: Becoming Leader. State: Replica: 35ef99a703104e308f4484173be191a2, State: Running, Role: LEADER
I20250628 07:55:08.957690 10366 consensus_queue.cc:237] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } }
I20250628 07:55:08.960402 9956 raft_consensus.cc:2466] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ef99a703104e308f4484173be191a2 in term 1.
I20250628 07:55:08.969777 9812 catalog_manager.cc:5582] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ef99a703104e308f4484173be191a2 (127.3.91.131). New cstate: current_term: 1 leader_uuid: "35ef99a703104e308f4484173be191a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ef99a703104e308f4484173be191a2" member_type: VOTER last_known_addr { host: "127.3.91.131" port: 39601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 } health_report { overall_health: UNKNOWN } } }
W20250628 07:55:09.061648 10273 tablet.cc:2378] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:55:09.085131 10137 tablet.cc:2378] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250628 07:55:09.151893 10002 tablet.cc:2378] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250628 07:55:09.158558 9956 raft_consensus.cc:1273] T 942734a979c44623bcb75e1cf80d6f87 P cf2eb810e5ce4cae9c0da0f9719ea127 [term 1 FOLLOWER]: Refusing update from remote peer 35ef99a703104e308f4484173be191a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 07:55:09.158669 10091 raft_consensus.cc:1273] T 942734a979c44623bcb75e1cf80d6f87 P 01ec04492fda409f8e9d8bed5a0aa036 [term 1 FOLLOWER]: Refusing update from remote peer 35ef99a703104e308f4484173be191a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250628 07:55:09.160383 10372 consensus_queue.cc:1035] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2eb810e5ce4cae9c0da0f9719ea127" member_type: VOTER last_known_addr { host: "127.3.91.129" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250628 07:55:09.161134 10366 consensus_queue.cc:1035] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ec04492fda409f8e9d8bed5a0aa036" member_type: VOTER last_known_addr { host: "127.3.91.130" port: 42269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250628 07:55:09.196066 10381 mvcc.cc:204] Tried to move back new op lower bound from 7172494578298224640 to 7172494577499488256. Current Snapshot: MvccSnapshot[applied={T|T < 7172494578298224640}]
I20250628 07:55:09.202399 10384 mvcc.cc:204] Tried to move back new op lower bound from 7172494578298224640 to 7172494577499488256. Current Snapshot: MvccSnapshot[applied={T|T < 7172494578298224640}]
I20250628 07:55:09.208628 10383 mvcc.cc:204] Tried to move back new op lower bound from 7172494578298224640 to 7172494577499488256. Current Snapshot: MvccSnapshot[applied={T|T < 7172494578298224640}]
I20250628 07:55:14.165635 10071 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250628 07:55:14.168902 10207 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250628 07:55:14.181741 9936 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
8985e35ad2e54225b68a9fa8b9ca7351 | 127.3.91.190:34247 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:44751 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
01ec04492fda409f8e9d8bed5a0aa036 | 127.3.91.130:42269 | HEALTHY | <none> | 0 | 0
35ef99a703104e308f4484173be191a2 | 127.3.91.131:39601 | HEALTHY | <none> | 1 | 0
cf2eb810e5ce4cae9c0da0f9719ea127 | 127.3.91.129:45525 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.3.91.129 | experimental | 127.3.91.129:45525
local_ip_for_outbound_sockets | 127.3.91.130 | experimental | 127.3.91.130:42269
local_ip_for_outbound_sockets | 127.3.91.131 | experimental | 127.3.91.131:39601
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-0/data/info.pb | hidden | 127.3.91.129:45525
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-1/data/info.pb | hidden | 127.3.91.130:42269
server_dump_info_path | /tmp/dist-test-task7GuWg7/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1751097128721223-3438-0/minicluster-data/ts-2/data/info.pb | hidden | 127.3.91.131:39601
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.3.91.148:44751 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250628 07:55:14.402233 3438 log_verifier.cc:126] Checking tablet 942734a979c44623bcb75e1cf80d6f87
I20250628 07:55:15.181963 3438 log_verifier.cc:177] Verified matching terms for 206 ops in tablet 942734a979c44623bcb75e1cf80d6f87
I20250628 07:55:15.183004 3438 log_verifier.cc:126] Checking tablet af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:15.183316 3438 log_verifier.cc:177] Verified matching terms for 0 ops in tablet af4f53efb52a4f6fa1d79a47d47e4de4
I20250628 07:55:15.208520 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9850
W20250628 07:55:15.253652 10159 connection.cc:537] client connection to 127.3.91.129:45525 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250628 07:55:15.254145 10159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250628 07:55:15.254594 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 10005
W20250628 07:55:15.264427 10159 consensus_peers.cc:489] T 942734a979c44623bcb75e1cf80d6f87 P 35ef99a703104e308f4484173be191a2 -> Peer cf2eb810e5ce4cae9c0da0f9719ea127 (127.3.91.129:45525): Couldn't send request to peer cf2eb810e5ce4cae9c0da0f9719ea127. Status: Network error: Client connection negotiation failed: client connection to 127.3.91.129:45525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250628 07:55:15.295696 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 10140
I20250628 07:55:15.337478 3438 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task7GuWg7/build/tsan/bin/kudu with pid 9780
2025-06-28T07:55:15Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37378 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37378 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (186609 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST