Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250813 07:52:36.550329 427 test_util.cc:276] Using random seed: 1890516542
W20250813 07:52:37.703075 427 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.108s user 0.496s sys 0.611s
W20250813 07:52:37.703496 427 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.108s user 0.496s sys 0.611s
I20250813 07:52:37.706086 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:52:37.706314 427 ts_itest-base.cc:116] --------------
I20250813 07:52:37.706497 427 ts_itest-base.cc:117] 4 tablet servers
I20250813 07:52:37.706688 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:52:37.706883 427 ts_itest-base.cc:119] --------------
2025-08-13T07:52:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:52:37Z Disabled control of system clock
I20250813 07:52:37.746457 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46431
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:46431 with env {}
W20250813 07:52:38.051281 441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:38.051862 441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:38.052307 441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:38.085726 441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:52:38.086038 441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:38.086284 441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:52:38.086516 441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:52:38.122225 441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:46431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:38.123517 441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:38.125151 441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:38.136653 447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:39.538766 446 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 441
W20250813 07:52:39.699509 441 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.563s user 0.592s sys 0.970s
W20250813 07:52:39.699923 441 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.563s user 0.592s sys 0.970s
W20250813 07:52:38.137118 448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:39.701859 450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:39.704417 449 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1563 milliseconds
I20250813 07:52:39.704479 441 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:39.705780 441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:39.708153 441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:39.709520 441 hybrid_clock.cc:648] HybridClock initialized: now 1755071559709448 us; error 71 us; skew 500 ppm
I20250813 07:52:39.710350 441 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:39.716529 441 webserver.cc:480] Webserver started at http://127.0.106.254:42487/ using document root <none> and password file <none>
I20250813 07:52:39.717412 441 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:39.717664 441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:39.718114 441 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:39.722764 441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a6007238efd64d8cb0281e1b63fc6fa6"
format_stamp: "Formatted at 2025-08-13 07:52:39 on dist-test-slave-dr1z"
I20250813 07:52:39.723870 441 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a6007238efd64d8cb0281e1b63fc6fa6"
format_stamp: "Formatted at 2025-08-13 07:52:39 on dist-test-slave-dr1z"
I20250813 07:52:39.730895 441 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250813 07:52:39.736620 457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:39.737694 441 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250813 07:52:39.738021 441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a6007238efd64d8cb0281e1b63fc6fa6"
format_stamp: "Formatted at 2025-08-13 07:52:39 on dist-test-slave-dr1z"
I20250813 07:52:39.738328 441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:39.788635 441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:39.790601 441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:39.791065 441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:39.870997 441 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:46431
I20250813 07:52:39.871089 508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:46431 every 8 connection(s)
I20250813 07:52:39.874780 441 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:52:39.880605 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 441
I20250813 07:52:39.880820 509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:39.881024 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:52:39.904873 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Bootstrap starting.
I20250813 07:52:39.912451 509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Neither blocks nor log segments found. Creating new log.
I20250813 07:52:39.914212 509 log.cc:826] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:39.919574 509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: No bootstrap required, opened a new log
I20250813 07:52:39.940029 509 raft_consensus.cc:357] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:39.940680 509 raft_consensus.cc:383] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:52:39.940919 509 raft_consensus.cc:738] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6007238efd64d8cb0281e1b63fc6fa6, State: Initialized, Role: FOLLOWER
I20250813 07:52:39.941615 509 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:39.942108 509 raft_consensus.cc:397] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:52:39.942373 509 raft_consensus.cc:491] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:52:39.942692 509 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:52:39.946671 509 raft_consensus.cc:513] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:39.947351 509 leader_election.cc:304] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6007238efd64d8cb0281e1b63fc6fa6; no voters:
I20250813 07:52:39.949553 509 leader_election.cc:290] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:52:39.950232 514 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:52:39.952250 514 raft_consensus.cc:695] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 LEADER]: Becoming Leader. State: Replica: a6007238efd64d8cb0281e1b63fc6fa6, State: Running, Role: LEADER
I20250813 07:52:39.953183 514 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:39.954401 509 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:52:39.965127 515 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } } }
I20250813 07:52:39.965538 516 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6007238efd64d8cb0281e1b63fc6fa6. Latest consensus state: current_term: 1 leader_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } } }
I20250813 07:52:39.966678 515 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:39.966673 516 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:39.970131 524 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:52:39.980935 524 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:52:39.998724 524 catalog_manager.cc:1349] Generated new cluster ID: 2b9c91b1034b4b888cf1dacb543128db
I20250813 07:52:39.999012 524 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:52:40.026839 524 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:52:40.028791 524 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:52:40.052104 524 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Generated new TSK 0
I20250813 07:52:40.053151 524 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:52:40.067170 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 07:52:40.373615 533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:40.374128 533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:40.374645 533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:40.406208 533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:40.407120 533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:52:40.441741 533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:40.443009 533 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:40.444545 533 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:40.457340 539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:40.458673 540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:41.839049 541 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1377 milliseconds
W20250813 07:52:41.842460 542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:41.843462 533 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.384s user 0.000s sys 0.009s
W20250813 07:52:41.843715 533 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.385s user 0.000s sys 0.009s
I20250813 07:52:41.843932 533 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:41.844924 533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:41.847313 533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:41.848757 533 hybrid_clock.cc:648] HybridClock initialized: now 1755071561848697 us; error 73 us; skew 500 ppm
I20250813 07:52:41.849581 533 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:41.857815 533 webserver.cc:480] Webserver started at http://127.0.106.193:39797/ using document root <none> and password file <none>
I20250813 07:52:41.858747 533 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:41.858960 533 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:41.859404 533 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:41.866003 533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3618899b9d3d46fb8b5056b029b3c490"
format_stamp: "Formatted at 2025-08-13 07:52:41 on dist-test-slave-dr1z"
I20250813 07:52:41.867595 533 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3618899b9d3d46fb8b5056b029b3c490"
format_stamp: "Formatted at 2025-08-13 07:52:41 on dist-test-slave-dr1z"
I20250813 07:52:41.883581 533 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.009s sys 0.004s
I20250813 07:52:41.890421 549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:41.891549 533 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250813 07:52:41.891873 533 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3618899b9d3d46fb8b5056b029b3c490"
format_stamp: "Formatted at 2025-08-13 07:52:41 on dist-test-slave-dr1z"
I20250813 07:52:41.892210 533 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:41.958169 533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:41.959667 533 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:41.960116 533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:41.963003 533 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:52:41.967823 533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:52:41.968041 533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:41.968317 533 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:52:41.968478 533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:42.142782 533 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:43535
I20250813 07:52:42.142959 661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:43535 every 8 connection(s)
I20250813 07:52:42.145843 533 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:52:42.150430 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 533
I20250813 07:52:42.150913 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:52:42.159832 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:52:42.185016 662 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:42.185570 662 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:42.186759 662 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:42.189740 474 ts_manager.cc:194] Registered new tserver with Master: 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193:43535)
I20250813 07:52:42.192982 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:59205
W20250813 07:52:42.473455 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:42.473932 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:42.474380 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:42.505584 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:42.506430 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:52:42.541538 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:42.542762 666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:42.544278 666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:42.556792 672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:52:43.197151 662 heartbeater.cc:499] Master 127.0.106.254:46431 was elected leader, sending a full tablet report...
W20250813 07:52:42.558223 673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:43.679747 675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:43.681723 674 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1120 milliseconds
I20250813 07:52:43.681808 666 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:43.682966 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:43.686040 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:43.687470 666 hybrid_clock.cc:648] HybridClock initialized: now 1755071563687425 us; error 52 us; skew 500 ppm
I20250813 07:52:43.688254 666 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:43.694378 666 webserver.cc:480] Webserver started at http://127.0.106.194:39805/ using document root <none> and password file <none>
I20250813 07:52:43.695251 666 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:43.695459 666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:43.695907 666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:43.700135 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "506d40901fc14a869728fdfba945fa62"
format_stamp: "Formatted at 2025-08-13 07:52:43 on dist-test-slave-dr1z"
I20250813 07:52:43.701198 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "506d40901fc14a869728fdfba945fa62"
format_stamp: "Formatted at 2025-08-13 07:52:43 on dist-test-slave-dr1z"
I20250813 07:52:43.708256 666 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250813 07:52:43.714385 682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:43.715453 666 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:52:43.715798 666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "506d40901fc14a869728fdfba945fa62"
format_stamp: "Formatted at 2025-08-13 07:52:43 on dist-test-slave-dr1z"
I20250813 07:52:43.716135 666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:43.784986 666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:43.786512 666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:43.786952 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:43.789898 666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:52:43.794061 666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:52:43.794289 666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:43.794541 666 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:52:43.794703 666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:43.953455 666 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:35541
I20250813 07:52:43.953619 794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:35541 every 8 connection(s)
I20250813 07:52:43.956079 666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:52:43.958801 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 666
I20250813 07:52:43.959267 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:52:43.966215 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:52:43.983621 795 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:43.984184 795 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:43.985589 795 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:43.988099 474 ts_manager.cc:194] Registered new tserver with Master: 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:43.989372 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:55135
W20250813 07:52:44.280009 799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:44.280515 799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:44.281001 799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:44.312716 799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:44.313812 799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:52:44.350560 799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:44.351858 799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:44.353685 799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:44.367524 805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:52:44.993510 795 heartbeater.cc:499] Master 127.0.106.254:46431 was elected leader, sending a full tablet report...
W20250813 07:52:44.368245 806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:45.479140 808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:45.480885 807 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1108 milliseconds
I20250813 07:52:45.480993 799 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:45.482182 799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:45.484659 799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:45.486083 799 hybrid_clock.cc:648] HybridClock initialized: now 1755071565486025 us; error 64 us; skew 500 ppm
I20250813 07:52:45.486896 799 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:45.493582 799 webserver.cc:480] Webserver started at http://127.0.106.195:40943/ using document root <none> and password file <none>
I20250813 07:52:45.494446 799 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:45.494637 799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:45.495034 799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:45.499393 799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "95ab33ce149143878cd9ee4b2c541530"
format_stamp: "Formatted at 2025-08-13 07:52:45 on dist-test-slave-dr1z"
I20250813 07:52:45.500655 799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "95ab33ce149143878cd9ee4b2c541530"
format_stamp: "Formatted at 2025-08-13 07:52:45 on dist-test-slave-dr1z"
I20250813 07:52:45.508083 799 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20250813 07:52:45.513520 815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:45.514526 799 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250813 07:52:45.514869 799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "95ab33ce149143878cd9ee4b2c541530"
format_stamp: "Formatted at 2025-08-13 07:52:45 on dist-test-slave-dr1z"
I20250813 07:52:45.515187 799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:45.559237 799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:45.560719 799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:45.561133 799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:45.563643 799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:52:45.567571 799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:52:45.567795 799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:45.568022 799 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:52:45.568174 799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:45.715541 799 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:44083
I20250813 07:52:45.715628 927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:44083 every 8 connection(s)
I20250813 07:52:45.718034 799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:52:45.724030 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 799
I20250813 07:52:45.724536 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:52:45.731205 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.196:0
--local_ip_for_outbound_sockets=127.0.106.196
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:52:45.740608 928 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:45.741042 928 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:45.742127 928 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:45.744249 474 ts_manager.cc:194] Registered new tserver with Master: 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083)
I20250813 07:52:45.745546 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:39973
W20250813 07:52:46.047217 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:46.047725 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:46.048254 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:46.080260 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:46.081142 932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.196
I20250813 07:52:46.117903 932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46431
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.196
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:46.119211 932 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:46.120805 932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:46.132990 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:52:46.748759 928 heartbeater.cc:499] Master 127.0.106.254:46431 was elected leader, sending a full tablet report...
W20250813 07:52:46.134382 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:47.241365 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:47.243646 940 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250813 07:52:47.243732 932 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:47.244850 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:47.247067 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:47.248449 932 hybrid_clock.cc:648] HybridClock initialized: now 1755071567248392 us; error 59 us; skew 500 ppm
I20250813 07:52:47.249248 932 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:47.256057 932 webserver.cc:480] Webserver started at http://127.0.106.196:35217/ using document root <none> and password file <none>
I20250813 07:52:47.257076 932 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:47.257277 932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:47.257748 932 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:47.262342 932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8d26bd8b2be345f2ba113d583f331a79"
format_stamp: "Formatted at 2025-08-13 07:52:47 on dist-test-slave-dr1z"
I20250813 07:52:47.263425 932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8d26bd8b2be345f2ba113d583f331a79"
format_stamp: "Formatted at 2025-08-13 07:52:47 on dist-test-slave-dr1z"
I20250813 07:52:47.271097 932 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250813 07:52:47.277395 949 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:47.278489 932 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:52:47.278906 932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8d26bd8b2be345f2ba113d583f331a79"
format_stamp: "Formatted at 2025-08-13 07:52:47 on dist-test-slave-dr1z"
I20250813 07:52:47.279284 932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:47.321399 932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:47.322794 932 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:47.323225 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:47.326097 932 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:52:47.329941 932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:52:47.330152 932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:47.330382 932 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:52:47.330531 932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:47.474488 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.196:43943
I20250813 07:52:47.474582 1061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.196:43943 every 8 connection(s)
I20250813 07:52:47.476930 932 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250813 07:52:47.480926 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 932
I20250813 07:52:47.481514 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250813 07:52:47.498790 1062 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:47.499284 1062 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:47.500283 1062 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:47.502794 474 ts_manager.cc:194] Registered new tserver with Master: 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943)
I20250813 07:52:47.503983 474 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.196:35899
I20250813 07:52:47.515391 427 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250813 07:52:47.557179 474 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60100:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 07:52:47.627164 863 tablet_service.cc:1468] Processing CreateTablet for tablet cbfee60aadd44a70bb0d9fefc90171ee (DEFAULT_TABLE table=TestTable [id=023448e551fe4b2ca25e5689b3650101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:52:47.629586 863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfee60aadd44a70bb0d9fefc90171ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:47.637682 730 tablet_service.cc:1468] Processing CreateTablet for tablet cbfee60aadd44a70bb0d9fefc90171ee (DEFAULT_TABLE table=TestTable [id=023448e551fe4b2ca25e5689b3650101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:52:47.639348 730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfee60aadd44a70bb0d9fefc90171ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:47.637738 997 tablet_service.cc:1468] Processing CreateTablet for tablet cbfee60aadd44a70bb0d9fefc90171ee (DEFAULT_TABLE table=TestTable [id=023448e551fe4b2ca25e5689b3650101]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:52:47.640493 997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfee60aadd44a70bb0d9fefc90171ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:47.658335 1081 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Bootstrap starting.
I20250813 07:52:47.666972 1082 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Bootstrap starting.
I20250813 07:52:47.667131 1081 tablet_bootstrap.cc:654] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Neither blocks nor log segments found. Creating new log.
I20250813 07:52:47.669903 1083 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Bootstrap starting.
I20250813 07:52:47.670902 1081 log.cc:826] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:47.674381 1082 tablet_bootstrap.cc:654] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Neither blocks nor log segments found. Creating new log.
I20250813 07:52:47.676800 1082 log.cc:826] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:47.677373 1081 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: No bootstrap required, opened a new log
I20250813 07:52:47.677883 1083 tablet_bootstrap.cc:654] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Neither blocks nor log segments found. Creating new log.
I20250813 07:52:47.677915 1081 ts_tablet_manager.cc:1397] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20250813 07:52:47.680526 1083 log.cc:826] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:47.681720 1082 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: No bootstrap required, opened a new log
I20250813 07:52:47.682183 1082 ts_tablet_manager.cc:1397] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250813 07:52:47.686643 1083 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: No bootstrap required, opened a new log
I20250813 07:52:47.687157 1083 ts_tablet_manager.cc:1397] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250813 07:52:47.701598 1082 raft_consensus.cc:357] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.702270 1082 raft_consensus.cc:383] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:52:47.703006 1082 raft_consensus.cc:738] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d40901fc14a869728fdfba945fa62, State: Initialized, Role: FOLLOWER
I20250813 07:52:47.703456 1081 raft_consensus.cc:357] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.703789 1082 consensus_queue.cc:260] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.704314 1081 raft_consensus.cc:383] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:52:47.704600 1081 raft_consensus.cc:738] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ab33ce149143878cd9ee4b2c541530, State: Initialized, Role: FOLLOWER
I20250813 07:52:47.705638 1081 consensus_queue.cc:260] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.707515 1082 ts_tablet_manager.cc:1428] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Time spent starting tablet: real 0.025s user 0.016s sys 0.008s
I20250813 07:52:47.709527 1081 ts_tablet_manager.cc:1428] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Time spent starting tablet: real 0.031s user 0.020s sys 0.010s
W20250813 07:52:47.714864 796 tablet.cc:2378] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:52:47.716675 1083 raft_consensus.cc:357] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.718120 1083 raft_consensus.cc:383] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:52:47.718439 1083 raft_consensus.cc:738] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d26bd8b2be345f2ba113d583f331a79, State: Initialized, Role: FOLLOWER
I20250813 07:52:47.719297 1083 consensus_queue.cc:260] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
W20250813 07:52:47.725033 929 tablet.cc:2378] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:52:47.728209 1062 heartbeater.cc:499] Master 127.0.106.254:46431 was elected leader, sending a full tablet report...
I20250813 07:52:47.728260 1083 ts_tablet_manager.cc:1428] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Time spent starting tablet: real 0.041s user 0.031s sys 0.008s
W20250813 07:52:47.731483 1063 tablet.cc:2378] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:52:47.747165 1089 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:52:47.747741 1089 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.750104 1089 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083), 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:47.760277 883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "8d26bd8b2be345f2ba113d583f331a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ab33ce149143878cd9ee4b2c541530" is_pre_election: true
I20250813 07:52:47.761032 883 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d26bd8b2be345f2ba113d583f331a79 in term 0.
I20250813 07:52:47.762256 952 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d26bd8b2be345f2ba113d583f331a79, 95ab33ce149143878cd9ee4b2c541530; no voters:
I20250813 07:52:47.762909 1089 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:52:47.763154 1089 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:52:47.762506 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "8d26bd8b2be345f2ba113d583f331a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62" is_pre_election: true
I20250813 07:52:47.763409 1089 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:52:47.763732 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d26bd8b2be345f2ba113d583f331a79 in term 0.
I20250813 07:52:47.767747 1089 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.769680 1089 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [CANDIDATE]: Term 1 election: Requested vote from peers 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083), 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:47.770449 883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "8d26bd8b2be345f2ba113d583f331a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95ab33ce149143878cd9ee4b2c541530"
I20250813 07:52:47.770660 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "8d26bd8b2be345f2ba113d583f331a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62"
I20250813 07:52:47.770917 883 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:52:47.771139 750 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:52:47.776037 883 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d26bd8b2be345f2ba113d583f331a79 in term 1.
I20250813 07:52:47.776454 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d26bd8b2be345f2ba113d583f331a79 in term 1.
I20250813 07:52:47.776983 952 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d26bd8b2be345f2ba113d583f331a79, 95ab33ce149143878cd9ee4b2c541530; no voters:
I20250813 07:52:47.777645 1089 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:52:47.779100 1089 raft_consensus.cc:695] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [term 1 LEADER]: Becoming Leader. State: Replica: 8d26bd8b2be345f2ba113d583f331a79, State: Running, Role: LEADER
I20250813 07:52:47.779836 1089 consensus_queue.cc:237] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:47.790833 473 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196). New cstate: current_term: 1 leader_uuid: "8d26bd8b2be345f2ba113d583f331a79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20250813 07:52:47.823015 427 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250813 07:52:47.826543 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 506d40901fc14a869728fdfba945fa62 to finish bootstrapping
I20250813 07:52:47.840903 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 95ab33ce149143878cd9ee4b2c541530 to finish bootstrapping
I20250813 07:52:47.851545 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8d26bd8b2be345f2ba113d583f331a79 to finish bootstrapping
I20250813 07:52:47.863565 427 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250813 07:52:47.868199 427 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "95ab33ce149143878cd9ee4b2c541530"
rpc_addresses {
host: "127.0.106.195"
port: 44083
}
}
ts_infos {
permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79"
rpc_addresses {
host: "127.0.106.196"
port: 43943
}
}
ts_infos {
permanent_uuid: "506d40901fc14a869728fdfba945fa62"
rpc_addresses {
host: "127.0.106.194"
port: 35541
}
}
I20250813 07:52:48.357110 1096 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:52:48.376734 1096 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 8d26bd8b2be345f2ba113d583f331a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:52:48.379742 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 932
W20250813 07:52:48.402105 683 connection.cc:537] server connection from 127.0.106.196:41467 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250813 07:52:48.402851 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 441
I20250813 07:52:48.427139 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46431
--webserver_interface=127.0.106.254
--webserver_port=42487
--builtin_ntp_servers=127.0.106.212:40687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:46431 with env {}
W20250813 07:52:48.736308 1106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:48.736876 1106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:48.737330 1106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:48.767985 1106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:52:48.768297 1106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:48.768548 1106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:52:48.768779 1106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:52:48.804533 1106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:40687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:46431
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=42487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:48.805892 1106 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:48.807502 1106 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:48.818323 1112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:49.240687 662 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:46431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:46431: connect: Connection refused (error 111)
W20250813 07:52:49.402585 795 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:46431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:46431: connect: Connection refused (error 111)
W20250813 07:52:49.404613 928 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:46431 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:46431: connect: Connection refused (error 111)
I20250813 07:52:49.872551 1121 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d26bd8b2be345f2ba113d583f331a79)
I20250813 07:52:49.873251 1121 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:49.878477 1121 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943), 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
W20250813 07:52:49.878670 816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
W20250813 07:52:49.887586 816 leader_election.cc:336] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943): Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
I20250813 07:52:49.902952 1125 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8d26bd8b2be345f2ba113d583f331a79)
I20250813 07:52:49.903682 1125 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:49.922200 1125 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083), 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943)
W20250813 07:52:49.924396 683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
I20250813 07:52:49.932468 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "95ab33ce149143878cd9ee4b2c541530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62" is_pre_election: true
I20250813 07:52:49.933082 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ab33ce149143878cd9ee4b2c541530 in term 1.
I20250813 07:52:49.934598 819 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 506d40901fc14a869728fdfba945fa62, 95ab33ce149143878cd9ee4b2c541530; no voters: 8d26bd8b2be345f2ba113d583f331a79
I20250813 07:52:49.935498 1121 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250813 07:52:49.935819 1121 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8d26bd8b2be345f2ba113d583f331a79)
I20250813 07:52:49.936136 1121 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 1 FOLLOWER]: Advancing to term 2
W20250813 07:52:49.936435 683 leader_election.cc:336] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943): Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
I20250813 07:52:49.942736 1121 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:49.945046 1121 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 election: Requested vote from peers 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943), 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:49.946527 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "95ab33ce149143878cd9ee4b2c541530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62"
I20250813 07:52:49.947058 750 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:52:49.953397 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ab33ce149143878cd9ee4b2c541530 in term 2.
W20250813 07:52:49.949713 816 leader_election.cc:336] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943): Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
I20250813 07:52:49.959841 819 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 506d40901fc14a869728fdfba945fa62, 95ab33ce149143878cd9ee4b2c541530; no voters: 8d26bd8b2be345f2ba113d583f331a79
I20250813 07:52:49.965981 1121 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:52:49.975032 883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "506d40901fc14a869728fdfba945fa62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "95ab33ce149143878cd9ee4b2c541530" is_pre_election: true
I20250813 07:52:49.983322 1121 raft_consensus.cc:695] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 LEADER]: Becoming Leader. State: Replica: 95ab33ce149143878cd9ee4b2c541530, State: Running, Role: LEADER
I20250813 07:52:49.984513 1121 consensus_queue.cc:237] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:49.999094 685 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 506d40901fc14a869728fdfba945fa62; no voters: 8d26bd8b2be345f2ba113d583f331a79, 95ab33ce149143878cd9ee4b2c541530
I20250813 07:52:50.000134 1125 raft_consensus.cc:2747] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250813 07:52:50.220479 1111 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1106
W20250813 07:52:50.275842 1111 kernel_stack_watchdog.cc:198] Thread 1106 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 07:52:50.276441 1106 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.458s user 0.580s sys 0.833s
W20250813 07:52:48.818828 1113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:50.276916 1106 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.459s user 0.581s sys 0.833s
W20250813 07:52:50.277189 1114 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1456 milliseconds
I20250813 07:52:50.278740 1106 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 07:52:50.278797 1115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:52:50.281985 1106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:50.284328 1106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:50.285701 1106 hybrid_clock.cc:648] HybridClock initialized: now 1755071570285636 us; error 69 us; skew 500 ppm
I20250813 07:52:50.286522 1106 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:50.292338 1106 webserver.cc:480] Webserver started at http://127.0.106.254:42487/ using document root <none> and password file <none>
I20250813 07:52:50.293213 1106 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:50.293443 1106 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:50.301180 1106 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250813 07:52:50.305369 1137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:50.306459 1106 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 07:52:50.306784 1106 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a6007238efd64d8cb0281e1b63fc6fa6"
format_stamp: "Formatted at 2025-08-13 07:52:39 on dist-test-slave-dr1z"
I20250813 07:52:50.308724 1106 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:50.371747 1106 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:50.373231 1106 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:50.373689 1106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:50.441934 1106 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:46431
I20250813 07:52:50.441999 1189 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:46431 every 8 connection(s)
I20250813 07:52:50.444600 1106 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:52:50.449011 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1106
I20250813 07:52:50.449509 427 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 95ab33ce149143878cd9ee4b2c541530
I20250813 07:52:50.454762 1190 sys_catalog.cc:263] Verifying existing consensus state
I20250813 07:52:50.459770 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Bootstrap starting.
I20250813 07:52:50.497316 1190 log.cc:826] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:50.520217 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:52:50.520996 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Bootstrap complete.
I20250813 07:52:50.544344 1190 raft_consensus.cc:357] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:50.546682 1190 raft_consensus.cc:738] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6007238efd64d8cb0281e1b63fc6fa6, State: Initialized, Role: FOLLOWER
I20250813 07:52:50.547458 1190 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:50.548019 1190 raft_consensus.cc:397] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:52:50.548288 1190 raft_consensus.cc:491] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:52:50.548633 1190 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:52:50.554543 1190 raft_consensus.cc:513] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:50.555270 1190 leader_election.cc:304] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6007238efd64d8cb0281e1b63fc6fa6; no voters:
I20250813 07:52:50.557439 1190 leader_election.cc:290] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 07:52:50.557734 1195 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:52:50.558984 750 raft_consensus.cc:1273] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Refusing update from remote peer 95ab33ce149143878cd9ee4b2c541530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 07:52:50.560911 1194 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 07:52:50.568675 1190 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:52:50.575973 1195 raft_consensus.cc:695] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [term 2 LEADER]: Becoming Leader. State: Replica: a6007238efd64d8cb0281e1b63fc6fa6, State: Running, Role: LEADER
I20250813 07:52:50.576985 1195 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } }
I20250813 07:52:50.598698 1196 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } } }
I20250813 07:52:50.599431 1196 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:50.600907 1197 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6007238efd64d8cb0281e1b63fc6fa6. Latest consensus state: current_term: 2 leader_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6007238efd64d8cb0281e1b63fc6fa6" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46431 } } }
I20250813 07:52:50.601524 1197 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:50.611948 1209 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:52:50.626248 795 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:50.639719 928 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
W20250813 07:52:50.654874 816 consensus_peers.cc:489] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 -> Peer 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943): Couldn't send request to peer 8d26bd8b2be345f2ba113d583f331a79. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 07:52:50.664697 1209 catalog_manager.cc:671] Loaded metadata for table TestTable [id=023448e551fe4b2ca25e5689b3650101]
I20250813 07:52:50.672036 1209 tablet_loader.cc:96] loaded metadata for tablet cbfee60aadd44a70bb0d9fefc90171ee (table TestTable [id=023448e551fe4b2ca25e5689b3650101])
I20250813 07:52:50.673503 1209 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:52:50.678200 1209 catalog_manager.cc:1261] Loaded cluster ID: 2b9c91b1034b4b888cf1dacb543128db
I20250813 07:52:50.678547 1209 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:52:50.685739 1209 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:52:50.690382 1209 catalog_manager.cc:5966] T 00000000000000000000000000000000 P a6007238efd64d8cb0281e1b63fc6fa6: Loaded TSK: 0
I20250813 07:52:50.691987 1209 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 07:52:50.814728 1192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:50.815316 1192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:50.846937 1192 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250813 07:52:51.288625 662 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46431
I20250813 07:52:51.294646 1155 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" instance_seqno: 1755071562103306) as {username='slave'} at 127.0.106.193:48061; Asking this server to re-register.
I20250813 07:52:51.296288 662 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:51.297410 662 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:51.300097 1154 ts_manager.cc:194] Registered new tserver with Master: 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193:43535)
I20250813 07:52:51.649720 1154 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" instance_seqno: 1755071565681679) as {username='slave'} at 127.0.106.195:50727; Asking this server to re-register.
I20250813 07:52:51.651933 928 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:51.654250 1154 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "506d40901fc14a869728fdfba945fa62" instance_seqno: 1755071563917272) as {username='slave'} at 127.0.106.194:59171; Asking this server to re-register.
I20250813 07:52:51.652619 928 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:51.656467 795 heartbeater.cc:461] Registering TS with master...
I20250813 07:52:51.657080 795 heartbeater.cc:507] Master 127.0.106.254:46431 requested a full tablet report, sending...
I20250813 07:52:51.659956 1155 ts_manager.cc:194] Registered new tserver with Master: 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:51.666926 1154 ts_manager.cc:194] Registered new tserver with Master: 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083)
I20250813 07:52:51.664409 1155 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 reported cstate change: term changed from 1 to 2, leader changed from 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196) to 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195). New cstate: current_term: 2 leader_uuid: "95ab33ce149143878cd9ee4b2c541530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } }
W20250813 07:52:52.230015 1192 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.344s user 0.433s sys 0.900s
W20250813 07:52:52.230369 1192 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.345s user 0.433s sys 0.900s
I20250813 07:52:52.283776 883 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "95ab33ce149143878cd9ee4b2c541530"
tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "506d40901fc14a869728fdfba945fa62"
}
peers {
permanent_uuid: "95ab33ce149143878cd9ee4b2c541530"
}
}
from {username='slave'} at 127.0.0.1:48096
W20250813 07:52:52.285122 883 raft_consensus.cc:2216] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true
I20250813 07:52:52.286298 883 raft_consensus.cc:3053] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 LEADER]: Stepping down as leader of term 2
I20250813 07:52:52.286602 883 raft_consensus.cc:738] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 95ab33ce149143878cd9ee4b2c541530, State: Running, Role: LEADER
I20250813 07:52:52.287282 883 consensus_queue.cc:260] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:52.288465 883 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:52:53.296993 1252 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 95ab33ce149143878cd9ee4b2c541530)
I20250813 07:52:53.297578 1252 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } }
I20250813 07:52:53.299118 1252 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083), 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943)
I20250813 07:52:53.300066 883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "506d40901fc14a869728fdfba945fa62" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "95ab33ce149143878cd9ee4b2c541530" is_pre_election: true
W20250813 07:52:53.304162 683 leader_election.cc:336] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196:43943): Network error: Client connection negotiation failed: client connection to 127.0.106.196:43943: connect: Connection refused (error 111)
I20250813 07:52:53.304462 683 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 506d40901fc14a869728fdfba945fa62; no voters: 8d26bd8b2be345f2ba113d583f331a79, 95ab33ce149143878cd9ee4b2c541530
I20250813 07:52:53.304997 1252 raft_consensus.cc:2747] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250813 07:52:53.797295 1257 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250813 07:52:53.797721 1257 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true
I20250813 07:52:53.798753 1257 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:53.800066 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "95ab33ce149143878cd9ee4b2c541530" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62" is_pre_election: true
I20250813 07:52:53.800577 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ab33ce149143878cd9ee4b2c541530 in term 2.
I20250813 07:52:53.801497 819 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 506d40901fc14a869728fdfba945fa62, 95ab33ce149143878cd9ee4b2c541530; no voters:
I20250813 07:52:53.802064 1257 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250813 07:52:53.802309 1257 raft_consensus.cc:491] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250813 07:52:53.802546 1257 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 3 FOLLOWER]: Advancing to term 4
I20250813 07:52:53.807261 1257 raft_consensus.cc:513] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true
I20250813 07:52:53.808275 1257 leader_election.cc:290] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 4 election: Requested vote from peers 506d40901fc14a869728fdfba945fa62 (127.0.106.194:35541)
I20250813 07:52:53.809206 750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee" candidate_uuid: "95ab33ce149143878cd9ee4b2c541530" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "506d40901fc14a869728fdfba945fa62"
I20250813 07:52:53.809623 750 raft_consensus.cc:3058] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 2 FOLLOWER]: Advancing to term 4
I20250813 07:52:53.814044 750 raft_consensus.cc:2466] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ab33ce149143878cd9ee4b2c541530 in term 4.
I20250813 07:52:53.814890 819 leader_election.cc:304] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 506d40901fc14a869728fdfba945fa62, 95ab33ce149143878cd9ee4b2c541530; no voters:
I20250813 07:52:53.815438 1257 raft_consensus.cc:2802] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 FOLLOWER]: Leader election won for term 4
I20250813 07:52:53.816195 1257 raft_consensus.cc:695] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 LEADER]: Becoming Leader. State: Replica: 95ab33ce149143878cd9ee4b2c541530, State: Running, Role: LEADER
I20250813 07:52:53.816740 1257 consensus_queue.cc:237] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true
I20250813 07:52:53.822134 1154 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195), VOTER 506d40901fc14a869728fdfba945fa62 (127.0.106.194). New cstate: current_term: 4 leader_uuid: "95ab33ce149143878cd9ee4b2c541530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d26bd8b2be345f2ba113d583f331a79" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 43943 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true }
I20250813 07:52:54.406807 750 raft_consensus.cc:1273] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Refusing update from remote peer 95ab33ce149143878cd9ee4b2c541530: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250813 07:52:54.408164 1265 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250813 07:52:54.417150 1268 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true }
I20250813 07:52:54.418079 750 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true }
I20250813 07:52:54.427783 1154 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 reported cstate change: config changed from index -1 to 3, VOTER 8d26bd8b2be345f2ba113d583f331a79 (127.0.106.196) evicted, no longer has a pending config: VOTER 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195), VOTER 506d40901fc14a869728fdfba945fa62 (127.0.106.194). New cstate: current_term: 4 leader_uuid: "95ab33ce149143878cd9ee4b2c541530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } unsafe_config_change: true }
W20250813 07:52:54.436607 1154 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet cbfee60aadd44a70bb0d9fefc90171ee on TS 8d26bd8b2be345f2ba113d583f331a79: Not found: failed to reset TS proxy: Could not find TS for UUID 8d26bd8b2be345f2ba113d583f331a79
I20250813 07:52:54.458839 883 consensus_queue.cc:237] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true
I20250813 07:52:54.465989 750 raft_consensus.cc:1273] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Refusing update from remote peer 95ab33ce149143878cd9ee4b2c541530: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250813 07:52:54.467299 1271 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250813 07:52:54.472707 1265 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true }
I20250813 07:52:54.474284 750 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true }
W20250813 07:52:54.476159 819 consensus_peers.cc:489] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 -> Peer 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193:43535): Couldn't send request to peer 3618899b9d3d46fb8b5056b029b3c490. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbfee60aadd44a70bb0d9fefc90171ee. This is attempt 1: this message will repeat every 5th retry.
I20250813 07:52:54.482106 1140 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cbfee60aadd44a70bb0d9fefc90171ee with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250813 07:52:54.484177 1154 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 reported cstate change: config changed from index 3 to 5, NON_VOTER 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) added. New cstate: current_term: 4 leader_uuid: "95ab33ce149143878cd9ee4b2c541530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true }
W20250813 07:52:54.489085 1139 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet cbfee60aadd44a70bb0d9fefc90171ee on TS 8d26bd8b2be345f2ba113d583f331a79 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8d26bd8b2be345f2ba113d583f331a79
I20250813 07:52:54.984493 1279 ts_tablet_manager.cc:927] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Initiating tablet copy from peer 95ab33ce149143878cd9ee4b2c541530 (127.0.106.195:44083)
I20250813 07:52:54.986547 1279 tablet_copy_client.cc:323] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.195:44083
I20250813 07:52:54.997085 903 tablet_copy_service.cc:140] P 95ab33ce149143878cd9ee4b2c541530: Received BeginTabletCopySession request for tablet cbfee60aadd44a70bb0d9fefc90171ee from peer 3618899b9d3d46fb8b5056b029b3c490 ({username='slave'} at 127.0.106.193:60717)
I20250813 07:52:54.997534 903 tablet_copy_service.cc:161] P 95ab33ce149143878cd9ee4b2c541530: Beginning new tablet copy session on tablet cbfee60aadd44a70bb0d9fefc90171ee from peer 3618899b9d3d46fb8b5056b029b3c490 at {username='slave'} at 127.0.106.193:60717: session id = 3618899b9d3d46fb8b5056b029b3c490-cbfee60aadd44a70bb0d9fefc90171ee
I20250813 07:52:55.002398 903 tablet_copy_source_session.cc:215] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 07:52:55.007122 1279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfee60aadd44a70bb0d9fefc90171ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:55.025323 1279 tablet_copy_client.cc:806] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: tablet copy: Starting download of 0 data blocks...
I20250813 07:52:55.025861 1279 tablet_copy_client.cc:670] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: tablet copy: Starting download of 1 WAL segments...
I20250813 07:52:55.029644 1279 tablet_copy_client.cc:538] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 07:52:55.035544 1279 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Bootstrap starting.
I20250813 07:52:55.047792 1279 log.cc:826] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:55.058586 1279 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:52:55.059422 1279 tablet_bootstrap.cc:492] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Bootstrap complete.
I20250813 07:52:55.060050 1279 ts_tablet_manager.cc:1397] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20250813 07:52:55.076702 1279 raft_consensus.cc:357] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true
I20250813 07:52:55.077605 1279 raft_consensus.cc:738] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 3618899b9d3d46fb8b5056b029b3c490, State: Initialized, Role: LEARNER
I20250813 07:52:55.078205 1279 consensus_queue.cc:260] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: true } } unsafe_config_change: true
I20250813 07:52:55.081640 1279 ts_tablet_manager.cc:1428] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490: Time spent starting tablet: real 0.021s user 0.015s sys 0.009s
I20250813 07:52:55.083396 903 tablet_copy_service.cc:342] P 95ab33ce149143878cd9ee4b2c541530: Request end of tablet copy session 3618899b9d3d46fb8b5056b029b3c490-cbfee60aadd44a70bb0d9fefc90171ee received from {username='slave'} at 127.0.106.193:60717
I20250813 07:52:55.083855 903 tablet_copy_service.cc:434] P 95ab33ce149143878cd9ee4b2c541530: ending tablet copy session 3618899b9d3d46fb8b5056b029b3c490-cbfee60aadd44a70bb0d9fefc90171ee on tablet cbfee60aadd44a70bb0d9fefc90171ee with peer 3618899b9d3d46fb8b5056b029b3c490
I20250813 07:52:55.528676 617 raft_consensus.cc:1215] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250813 07:52:55.656383 1139 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet cbfee60aadd44a70bb0d9fefc90171ee on TS 8d26bd8b2be345f2ba113d583f331a79 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8d26bd8b2be345f2ba113d583f331a79
I20250813 07:52:55.935956 1286 raft_consensus.cc:1062] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530: attempting to promote NON_VOTER 3618899b9d3d46fb8b5056b029b3c490 to VOTER
I20250813 07:52:55.937425 1286 consensus_queue.cc:237] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false } } unsafe_config_change: true
I20250813 07:52:55.941951 750 raft_consensus.cc:1273] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Refusing update from remote peer 95ab33ce149143878cd9ee4b2c541530: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250813 07:52:55.942188 617 raft_consensus.cc:1273] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [term 4 LEARNER]: Refusing update from remote peer 95ab33ce149143878cd9ee4b2c541530: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250813 07:52:55.942930 1286 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250813 07:52:55.943653 1285 consensus_queue.cc:1035] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250813 07:52:55.949316 1286 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false } } unsafe_config_change: true }
I20250813 07:52:55.950738 750 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 506d40901fc14a869728fdfba945fa62 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false } } unsafe_config_change: true }
I20250813 07:52:55.951733 617 raft_consensus.cc:2953] T cbfee60aadd44a70bb0d9fefc90171ee P 3618899b9d3d46fb8b5056b029b3c490 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false } } unsafe_config_change: true }
I20250813 07:52:55.961347 1154 catalog_manager.cc:5582] T cbfee60aadd44a70bb0d9fefc90171ee P 95ab33ce149143878cd9ee4b2c541530 reported cstate change: config changed from index 5 to 6, 3618899b9d3d46fb8b5056b029b3c490 (127.0.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "95ab33ce149143878cd9ee4b2c541530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "95ab33ce149143878cd9ee4b2c541530" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506d40901fc14a869728fdfba945fa62" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43535 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250813 07:52:55.970093 427 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250813 07:52:55.996785 427 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "cbfee60aadd44a70bb0d9fefc90171ee"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "95ab33ce149143878cd9ee4b2c541530"
rpc_addresses {
host: "127.0.106.195"
port: 44083
}
}
ts_infos {
permanent_uuid: "506d40901fc14a869728fdfba945fa62"
rpc_addresses {
host: "127.0.106.194"
port: 35541
}
}
ts_infos {
permanent_uuid: "3618899b9d3d46fb8b5056b029b3c490"
rpc_addresses {
host: "127.0.106.193"
port: 43535
}
}
I20250813 07:52:55.998960 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 533
I20250813 07:52:56.018471 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 666
I20250813 07:52:56.037940 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 799
I20250813 07:52:56.060065 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1106
2025-08-13T07:52:56Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19564 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250813 07:52:56.112494 427 test_util.cc:276] Using random seed: 1910078806
I20250813 07:52:56.118155 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:52:56.118319 427 ts_itest-base.cc:116] --------------
I20250813 07:52:56.118474 427 ts_itest-base.cc:117] 3 tablet servers
I20250813 07:52:56.118608 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:52:56.118737 427 ts_itest-base.cc:119] --------------
2025-08-13T07:52:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:52:56Z Disabled control of system clock
I20250813 07:52:56.152482 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:38893
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:38893
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250813 07:52:56.440598 1304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:56.441115 1304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:56.441555 1304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:56.472395 1304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:52:56.472677 1304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:56.472879 1304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:52:56.473074 1304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:52:56.507575 1304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:38893
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:38893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:56.508786 1304 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:56.510385 1304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:56.520323 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:56.521901 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:57.635911 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:57.638116 1312 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1112 milliseconds
I20250813 07:52:57.638229 1304 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:57.639376 1304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:57.642298 1304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:57.643771 1304 hybrid_clock.cc:648] HybridClock initialized: now 1755071577643732 us; error 49 us; skew 500 ppm
I20250813 07:52:57.644554 1304 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:57.650687 1304 webserver.cc:480] Webserver started at http://127.0.106.254:43901/ using document root <none> and password file <none>
I20250813 07:52:57.651573 1304 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:57.651763 1304 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:57.652185 1304 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:57.656415 1304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "30629ff3c598423381268ab4d794dc23"
format_stamp: "Formatted at 2025-08-13 07:52:57 on dist-test-slave-dr1z"
I20250813 07:52:57.657459 1304 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "30629ff3c598423381268ab4d794dc23"
format_stamp: "Formatted at 2025-08-13 07:52:57 on dist-test-slave-dr1z"
I20250813 07:52:57.664510 1304 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250813 07:52:57.669852 1321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:57.670878 1304 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250813 07:52:57.671185 1304 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "30629ff3c598423381268ab4d794dc23"
format_stamp: "Formatted at 2025-08-13 07:52:57 on dist-test-slave-dr1z"
I20250813 07:52:57.671506 1304 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:57.715610 1304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:57.717025 1304 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:57.717448 1304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:57.783869 1304 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:38893
I20250813 07:52:57.783950 1372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:38893 every 8 connection(s)
I20250813 07:52:57.786513 1304 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:52:57.791383 1373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:52:57.796409 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1304
I20250813 07:52:57.796880 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:52:57.814980 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23: Bootstrap starting.
I20250813 07:52:57.821333 1373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23: Neither blocks nor log segments found. Creating new log.
I20250813 07:52:57.822989 1373 log.cc:826] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23: Log is configured to *not* fsync() on all Append() calls
I20250813 07:52:57.827440 1373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23: No bootstrap required, opened a new log
I20250813 07:52:57.843925 1373 raft_consensus.cc:357] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } }
I20250813 07:52:57.844568 1373 raft_consensus.cc:383] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:52:57.844769 1373 raft_consensus.cc:738] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30629ff3c598423381268ab4d794dc23, State: Initialized, Role: FOLLOWER
I20250813 07:52:57.845371 1373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } }
I20250813 07:52:57.845883 1373 raft_consensus.cc:397] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:52:57.846103 1373 raft_consensus.cc:491] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:52:57.846376 1373 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:52:57.850229 1373 raft_consensus.cc:513] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } }
I20250813 07:52:57.850888 1373 leader_election.cc:304] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30629ff3c598423381268ab4d794dc23; no voters:
I20250813 07:52:57.852427 1373 leader_election.cc:290] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:52:57.853135 1378 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:52:57.855136 1378 raft_consensus.cc:695] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [term 1 LEADER]: Becoming Leader. State: Replica: 30629ff3c598423381268ab4d794dc23, State: Running, Role: LEADER
I20250813 07:52:57.855801 1378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } }
I20250813 07:52:57.856220 1373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:52:57.865800 1380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30629ff3c598423381268ab4d794dc23. Latest consensus state: current_term: 1 leader_uuid: "30629ff3c598423381268ab4d794dc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } } }
I20250813 07:52:57.866640 1380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:57.867192 1379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30629ff3c598423381268ab4d794dc23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30629ff3c598423381268ab4d794dc23" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 38893 } } }
I20250813 07:52:57.868047 1379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23 [sys.catalog]: This master's current role is: LEADER
I20250813 07:52:57.873519 1385 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:52:57.884104 1385 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:52:57.899348 1385 catalog_manager.cc:1349] Generated new cluster ID: 6ff00e9b5e69491fb181da157595a79c
I20250813 07:52:57.899627 1385 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:52:57.921838 1385 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:52:57.923373 1385 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:52:57.944928 1385 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 30629ff3c598423381268ab4d794dc23: Generated new TSK 0
I20250813 07:52:57.946080 1385 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:52:57.960228 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--builtin_ntp_servers=127.0.106.212:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250813 07:52:58.259440 1397 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 07:52:58.260067 1397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:52:58.260317 1397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:52:58.260797 1397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:52:58.292709 1397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:52:58.293668 1397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:52:58.330480 1397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:52:58.331806 1397 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:52:58.333413 1397 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:52:58.345527 1403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:59.746760 1402 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 1397
W20250813 07:52:59.768090 1402 kernel_stack_watchdog.cc:198] Thread 1397 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 07:52:58.347185 1404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:59.769213 1405 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1421 milliseconds
W20250813 07:52:59.768999 1397 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.423s user 0.533s sys 0.848s
W20250813 07:52:59.770134 1406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:52:59.770486 1397 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.425s user 0.534s sys 0.848s
I20250813 07:52:59.770797 1397 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:52:59.773546 1397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:52:59.776057 1397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:52:59.777516 1397 hybrid_clock.cc:648] HybridClock initialized: now 1755071579777449 us; error 49 us; skew 500 ppm
I20250813 07:52:59.778549 1397 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:52:59.786034 1397 webserver.cc:480] Webserver started at http://127.0.106.193:35491/ using document root <none> and password file <none>
I20250813 07:52:59.787269 1397 fs_manager.cc:362] Metadata directory not provided
I20250813 07:52:59.787540 1397 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:52:59.788096 1397 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:52:59.795969 1397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
format_stamp: "Formatted at 2025-08-13 07:52:59 on dist-test-slave-dr1z"
I20250813 07:52:59.797448 1397 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
format_stamp: "Formatted at 2025-08-13 07:52:59 on dist-test-slave-dr1z"
I20250813 07:52:59.806499 1397 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250813 07:52:59.813588 1413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:59.814725 1397 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:52:59.815109 1397 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
format_stamp: "Formatted at 2025-08-13 07:52:59 on dist-test-slave-dr1z"
I20250813 07:52:59.815532 1397 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:52:59.884399 1397 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:52:59.886339 1397 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:52:59.886901 1397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:52:59.890122 1397 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:52:59.895725 1397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:52:59.896011 1397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:52:59.896327 1397 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:52:59.896543 1397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:00.044056 1397 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:40211
I20250813 07:53:00.044121 1525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:40211 every 8 connection(s)
I20250813 07:53:00.047797 1397 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:53:00.050561 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1397
I20250813 07:53:00.051040 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:53:00.059934 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--builtin_ntp_servers=127.0.106.212:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250813 07:53:00.076629 1526 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38893
I20250813 07:53:00.077234 1526 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:00.078758 1526 heartbeater.cc:507] Master 127.0.106.254:38893 requested a full tablet report, sending...
I20250813 07:53:00.082062 1338 ts_manager.cc:194] Registered new tserver with Master: 0c1c9c4fb6dc4b86b4f2bdf0af31433b (127.0.106.193:40211)
I20250813 07:53:00.084797 1338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:52877
W20250813 07:53:00.358578 1530 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 07:53:00.359184 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:00.359441 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:00.359905 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:00.391088 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:00.391952 1530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:53:00.426442 1530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:00.427743 1530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:00.429370 1530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:00.440618 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:01.089048 1526 heartbeater.cc:499] Master 127.0.106.254:38893 was elected leader, sending a full tablet report...
W20250813 07:53:00.441727 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:01.635258 1538 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1188 milliseconds
W20250813 07:53:01.635943 1530 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.195s user 0.447s sys 0.736s
W20250813 07:53:01.636286 1530 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.195s user 0.447s sys 0.736s
W20250813 07:53:01.636626 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:01.636667 1530 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:01.638074 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:01.640273 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:01.641667 1530 hybrid_clock.cc:648] HybridClock initialized: now 1755071581641619 us; error 57 us; skew 500 ppm
I20250813 07:53:01.642494 1530 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:01.649873 1530 webserver.cc:480] Webserver started at http://127.0.106.194:45565/ using document root <none> and password file <none>
I20250813 07:53:01.650952 1530 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:01.651175 1530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:01.651609 1530 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:01.656167 1530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9f30c4f27b3f41b0b7134540aed584cf"
format_stamp: "Formatted at 2025-08-13 07:53:01 on dist-test-slave-dr1z"
I20250813 07:53:01.657285 1530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9f30c4f27b3f41b0b7134540aed584cf"
format_stamp: "Formatted at 2025-08-13 07:53:01 on dist-test-slave-dr1z"
I20250813 07:53:01.665584 1530 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250813 07:53:01.671720 1546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:01.673033 1530 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250813 07:53:01.673406 1530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9f30c4f27b3f41b0b7134540aed584cf"
format_stamp: "Formatted at 2025-08-13 07:53:01 on dist-test-slave-dr1z"
I20250813 07:53:01.673820 1530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:01.744604 1530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:01.746122 1530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:01.746560 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:01.749076 1530 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:01.753088 1530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:01.753309 1530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:01.753562 1530 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:01.753732 1530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:01.884270 1530 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:36387
I20250813 07:53:01.884366 1658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:36387 every 8 connection(s)
I20250813 07:53:01.886765 1530 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:53:01.893582 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1530
I20250813 07:53:01.894052 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:53:01.900512 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--builtin_ntp_servers=127.0.106.212:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250813 07:53:01.912819 1659 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38893
I20250813 07:53:01.913244 1659 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:01.914340 1659 heartbeater.cc:507] Master 127.0.106.254:38893 requested a full tablet report, sending...
I20250813 07:53:01.916653 1338 ts_manager.cc:194] Registered new tserver with Master: 9f30c4f27b3f41b0b7134540aed584cf (127.0.106.194:36387)
I20250813 07:53:01.917949 1338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:55055
W20250813 07:53:02.195466 1663 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 07:53:02.196085 1663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:02.196331 1663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:02.196796 1663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:02.227756 1663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:02.228592 1663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:53:02.263445 1663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:38893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:02.264736 1663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:02.266306 1663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:02.278574 1669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:02.921051 1659 heartbeater.cc:499] Master 127.0.106.254:38893 was elected leader, sending a full tablet report...
W20250813 07:53:02.278980 1670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:03.530478 1671 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1250 milliseconds
W20250813 07:53:03.531556 1672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:03.533960 1663 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.256s user 0.517s sys 0.731s
W20250813 07:53:03.534322 1663 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.256s user 0.517s sys 0.732s
I20250813 07:53:03.534590 1663 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:03.535975 1663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:03.538549 1663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:03.539984 1663 hybrid_clock.cc:648] HybridClock initialized: now 1755071583539959 us; error 44 us; skew 500 ppm
I20250813 07:53:03.540766 1663 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:03.548055 1663 webserver.cc:480] Webserver started at http://127.0.106.195:33009/ using document root <none> and password file <none>
I20250813 07:53:03.548992 1663 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:03.549218 1663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:03.549697 1663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:03.554147 1663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3a208f955ed84cee878476582869efe5"
format_stamp: "Formatted at 2025-08-13 07:53:03 on dist-test-slave-dr1z"
I20250813 07:53:03.555524 1663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3a208f955ed84cee878476582869efe5"
format_stamp: "Formatted at 2025-08-13 07:53:03 on dist-test-slave-dr1z"
I20250813 07:53:03.563726 1663 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250813 07:53:03.570088 1679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:03.571352 1663 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250813 07:53:03.571678 1663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3a208f955ed84cee878476582869efe5"
format_stamp: "Formatted at 2025-08-13 07:53:03 on dist-test-slave-dr1z"
I20250813 07:53:03.572031 1663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:03.661947 1663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:03.663450 1663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:03.663880 1663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:03.666393 1663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:03.670353 1663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:03.670571 1663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:03.670812 1663 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:03.670984 1663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:03.802655 1663 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:33081
I20250813 07:53:03.802757 1791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:33081 every 8 connection(s)
I20250813 07:53:03.805126 1663 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:53:03.811326 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1663
I20250813 07:53:03.811761 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:53:03.824684 1792 heartbeater.cc:344] Connected to a master server at 127.0.106.254:38893
I20250813 07:53:03.825111 1792 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:03.826093 1792 heartbeater.cc:507] Master 127.0.106.254:38893 requested a full tablet report, sending...
I20250813 07:53:03.828038 1338 ts_manager.cc:194] Registered new tserver with Master: 3a208f955ed84cee878476582869efe5 (127.0.106.195:33081)
I20250813 07:53:03.829501 1338 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:42337
I20250813 07:53:03.831439 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:03.868853 1337 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35174:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 07:53:03.887310 1337 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:53:03.940891 1461 tablet_service.cc:1468] Processing CreateTablet for tablet a6620e9bf1514dddba8f7919b4cbcb57 (DEFAULT_TABLE table=TestTable [id=db96cae757804b1caa10236676a3ae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:03.942802 1461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6620e9bf1514dddba8f7919b4cbcb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:03.945096 1727 tablet_service.cc:1468] Processing CreateTablet for tablet a6620e9bf1514dddba8f7919b4cbcb57 (DEFAULT_TABLE table=TestTable [id=db96cae757804b1caa10236676a3ae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:03.945853 1594 tablet_service.cc:1468] Processing CreateTablet for tablet a6620e9bf1514dddba8f7919b4cbcb57 (DEFAULT_TABLE table=TestTable [id=db96cae757804b1caa10236676a3ae95]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:03.947708 1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6620e9bf1514dddba8f7919b4cbcb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:03.947925 1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6620e9bf1514dddba8f7919b4cbcb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:03.968426 1811 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Bootstrap starting.
I20250813 07:53:03.974754 1811 tablet_bootstrap.cc:654] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:03.975937 1812 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Bootstrap starting.
I20250813 07:53:03.977527 1813 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Bootstrap starting.
I20250813 07:53:03.980444 1811 log.cc:826] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:03.983363 1812 tablet_bootstrap.cc:654] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:03.983736 1813 tablet_bootstrap.cc:654] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:03.985425 1813 log.cc:826] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:03.985563 1812 log.cc:826] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:03.986212 1811 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: No bootstrap required, opened a new log
I20250813 07:53:03.986734 1811 ts_tablet_manager.cc:1397] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.009s
I20250813 07:53:03.990239 1813 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: No bootstrap required, opened a new log
I20250813 07:53:03.990728 1813 ts_tablet_manager.cc:1397] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250813 07:53:03.992396 1812 tablet_bootstrap.cc:492] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: No bootstrap required, opened a new log
I20250813 07:53:03.992977 1812 ts_tablet_manager.cc:1397] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20250813 07:53:04.006834 1811 raft_consensus.cc:357] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.007561 1811 raft_consensus.cc:738] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1c9c4fb6dc4b86b4f2bdf0af31433b, State: Initialized, Role: FOLLOWER
I20250813 07:53:04.008167 1811 consensus_queue.cc:260] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.014097 1811 ts_tablet_manager.cc:1428] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Time spent starting tablet: real 0.027s user 0.018s sys 0.008s
I20250813 07:53:04.016445 1813 raft_consensus.cc:357] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.017184 1813 raft_consensus.cc:738] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a208f955ed84cee878476582869efe5, State: Initialized, Role: FOLLOWER
I20250813 07:53:04.017781 1813 consensus_queue.cc:260] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.020855 1792 heartbeater.cc:499] Master 127.0.106.254:38893 was elected leader, sending a full tablet report...
I20250813 07:53:04.020452 1812 raft_consensus.cc:357] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.021404 1812 raft_consensus.cc:738] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f30c4f27b3f41b0b7134540aed584cf, State: Initialized, Role: FOLLOWER
I20250813 07:53:04.022146 1813 ts_tablet_manager.cc:1428] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Time spent starting tablet: real 0.031s user 0.027s sys 0.005s
I20250813 07:53:04.022182 1812 consensus_queue.cc:260] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.026319 1812 ts_tablet_manager.cc:1428] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Time spent starting tablet: real 0.033s user 0.021s sys 0.009s
I20250813 07:53:04.042850 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:04.046830 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0c1c9c4fb6dc4b86b4f2bdf0af31433b to finish bootstrapping
W20250813 07:53:04.060418 1793 tablet.cc:2378] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:04.062175 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f30c4f27b3f41b0b7134540aed584cf to finish bootstrapping
W20250813 07:53:04.063741 1527 tablet.cc:2378] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:04.073997 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3a208f955ed84cee878476582869efe5 to finish bootstrapping
I20250813 07:53:04.114796 1481 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57"
dest_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
from {username='slave'} at 127.0.0.1:59766
I20250813 07:53:04.115365 1481 raft_consensus.cc:491] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250813 07:53:04.115665 1481 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:04.120002 1481 raft_consensus.cc:513] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.122328 1481 leader_election.cc:290] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [CANDIDATE]: Term 1 election: Requested vote from peers 9f30c4f27b3f41b0b7134540aed584cf (127.0.106.194:36387), 3a208f955ed84cee878476582869efe5 (127.0.106.195:33081)
I20250813 07:53:04.130700 427 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250813 07:53:04.134086 1747 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57" candidate_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a208f955ed84cee878476582869efe5"
I20250813 07:53:04.134430 1614 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57" candidate_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f30c4f27b3f41b0b7134540aed584cf"
I20250813 07:53:04.134666 1747 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:04.135059 1614 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:04.139027 1747 raft_consensus.cc:2466] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1c9c4fb6dc4b86b4f2bdf0af31433b in term 1.
I20250813 07:53:04.139247 1614 raft_consensus.cc:2466] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c1c9c4fb6dc4b86b4f2bdf0af31433b in term 1.
I20250813 07:53:04.140164 1416 leader_election.cc:304] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1c9c4fb6dc4b86b4f2bdf0af31433b, 3a208f955ed84cee878476582869efe5; no voters:
I20250813 07:53:04.140941 1817 raft_consensus.cc:2802] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:04.142442 1817 raft_consensus.cc:695] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 LEADER]: Becoming Leader. State: Replica: 0c1c9c4fb6dc4b86b4f2bdf0af31433b, State: Running, Role: LEADER
W20250813 07:53:04.143011 1660 tablet.cc:2378] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:04.143321 1817 consensus_queue.cc:237] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:04.154265 1336 catalog_manager.cc:5582] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c1c9c4fb6dc4b86b4f2bdf0af31433b (127.0.106.193). New cstate: current_term: 1 leader_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } health_report { overall_health: UNKNOWN } } }
I20250813 07:53:04.236423 427 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250813 07:53:04.441902 427 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250813 07:53:04.553979 1817 consensus_queue.cc:1035] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:53:04.569193 1824 consensus_queue.cc:1035] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:53:06.312803 1481 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57"
dest_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59776
I20250813 07:53:06.313306 1481 raft_consensus.cc:604] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 LEADER]: Received request to transfer leadership
I20250813 07:53:06.863613 1860 raft_consensus.cc:991] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b: : Instructing follower 3a208f955ed84cee878476582869efe5 to start an election
I20250813 07:53:06.863983 1859 raft_consensus.cc:1079] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 LEADER]: Signalling peer 3a208f955ed84cee878476582869efe5 to start an election
I20250813 07:53:06.865520 1747 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57"
dest_uuid: "3a208f955ed84cee878476582869efe5"
from {username='slave'} at 127.0.106.193:41967
I20250813 07:53:06.866019 1747 raft_consensus.cc:491] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250813 07:53:06.866284 1747 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:53:06.870388 1747 raft_consensus.cc:513] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:06.872496 1747 leader_election.cc:290] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [CANDIDATE]: Term 2 election: Requested vote from peers 0c1c9c4fb6dc4b86b4f2bdf0af31433b (127.0.106.193:40211), 9f30c4f27b3f41b0b7134540aed584cf (127.0.106.194:36387)
I20250813 07:53:06.885757 1481 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57" candidate_uuid: "3a208f955ed84cee878476582869efe5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
I20250813 07:53:06.885967 1614 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57" candidate_uuid: "3a208f955ed84cee878476582869efe5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f30c4f27b3f41b0b7134540aed584cf"
I20250813 07:53:06.886447 1614 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:53:06.886436 1481 raft_consensus.cc:3053] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 LEADER]: Stepping down as leader of term 1
I20250813 07:53:06.886739 1481 raft_consensus.cc:738] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0c1c9c4fb6dc4b86b4f2bdf0af31433b, State: Running, Role: LEADER
I20250813 07:53:06.887369 1481 consensus_queue.cc:260] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:06.888566 1481 raft_consensus.cc:3058] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:53:06.892634 1614 raft_consensus.cc:2466] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a208f955ed84cee878476582869efe5 in term 2.
I20250813 07:53:06.894035 1680 leader_election.cc:304] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a208f955ed84cee878476582869efe5, 9f30c4f27b3f41b0b7134540aed584cf; no voters:
I20250813 07:53:06.894572 1481 raft_consensus.cc:2466] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a208f955ed84cee878476582869efe5 in term 2.
I20250813 07:53:06.896193 1865 raft_consensus.cc:2802] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:53:06.897728 1865 raft_consensus.cc:695] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [term 2 LEADER]: Becoming Leader. State: Replica: 3a208f955ed84cee878476582869efe5, State: Running, Role: LEADER
I20250813 07:53:06.898619 1865 consensus_queue.cc:237] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } }
I20250813 07:53:06.906904 1335 catalog_manager.cc:5582] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 reported cstate change: term changed from 1 to 2, leader changed from 0c1c9c4fb6dc4b86b4f2bdf0af31433b (127.0.106.193) to 3a208f955ed84cee878476582869efe5 (127.0.106.195). New cstate: current_term: 2 leader_uuid: "3a208f955ed84cee878476582869efe5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a208f955ed84cee878476582869efe5" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33081 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:07.282259 1614 raft_consensus.cc:1273] T a6620e9bf1514dddba8f7919b4cbcb57 P 9f30c4f27b3f41b0b7134540aed584cf [term 2 FOLLOWER]: Refusing update from remote peer 3a208f955ed84cee878476582869efe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 07:53:07.283366 1865 consensus_queue.cc:1035] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f30c4f27b3f41b0b7134540aed584cf" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 07:53:07.294870 1481 raft_consensus.cc:1273] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 2 FOLLOWER]: Refusing update from remote peer 3a208f955ed84cee878476582869efe5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 07:53:07.296214 1865 consensus_queue.cc:1035] T a6620e9bf1514dddba8f7919b4cbcb57 P 3a208f955ed84cee878476582869efe5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 40211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 07:53:09.024411 1481 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a6620e9bf1514dddba8f7919b4cbcb57"
dest_uuid: "0c1c9c4fb6dc4b86b4f2bdf0af31433b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59786
I20250813 07:53:09.025075 1481 raft_consensus.cc:604] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 2 FOLLOWER]: Received request to transfer leadership
I20250813 07:53:09.025388 1481 raft_consensus.cc:612] T a6620e9bf1514dddba8f7919b4cbcb57 P 0c1c9c4fb6dc4b86b4f2bdf0af31433b [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250813 07:53:10.060376 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1397
I20250813 07:53:10.081180 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1530
I20250813 07:53:10.102386 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1663
I20250813 07:53:10.126456 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1304
2025-08-13T07:53:10Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14064 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250813 07:53:10.176812 427 test_util.cc:276] Using random seed: 1924143121
I20250813 07:53:10.181079 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:53:10.181239 427 ts_itest-base.cc:116] --------------
I20250813 07:53:10.181384 427 ts_itest-base.cc:117] 3 tablet servers
I20250813 07:53:10.181567 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:53:10.181720 427 ts_itest-base.cc:119] --------------
2025-08-13T07:53:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:53:10Z Disabled control of system clock
I20250813 07:53:10.215826 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:41369
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:41369 with env {}
W20250813 07:53:10.507081 1908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:10.507692 1908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:10.508184 1908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:10.539697 1908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:53:10.540037 1908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:10.540302 1908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:53:10.540535 1908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:53:10.576181 1908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:41369
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:10.577522 1908 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:10.579180 1908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:10.590310 1914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:10.590652 1915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:11.715587 1916 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1120 milliseconds
W20250813 07:53:11.715646 1908 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.126s user 0.409s sys 0.714s
W20250813 07:53:11.716112 1908 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.126s user 0.410s sys 0.715s
W20250813 07:53:11.717085 1917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:11.717164 1908 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:11.718362 1908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:11.720942 1908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:11.722328 1908 hybrid_clock.cc:648] HybridClock initialized: now 1755071591722299 us; error 36 us; skew 500 ppm
I20250813 07:53:11.723170 1908 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:11.730382 1908 webserver.cc:480] Webserver started at http://127.0.106.254:36443/ using document root <none> and password file <none>
I20250813 07:53:11.731339 1908 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:11.731556 1908 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:11.731995 1908 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:11.736373 1908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "98166fc4ec89470c91f605ff0e7eafbe"
format_stamp: "Formatted at 2025-08-13 07:53:11 on dist-test-slave-dr1z"
I20250813 07:53:11.737519 1908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "98166fc4ec89470c91f605ff0e7eafbe"
format_stamp: "Formatted at 2025-08-13 07:53:11 on dist-test-slave-dr1z"
I20250813 07:53:11.745208 1908 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250813 07:53:11.751087 1924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:11.752326 1908 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250813 07:53:11.752641 1908 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "98166fc4ec89470c91f605ff0e7eafbe"
format_stamp: "Formatted at 2025-08-13 07:53:11 on dist-test-slave-dr1z"
I20250813 07:53:11.752997 1908 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:11.816599 1908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:11.818178 1908 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:11.818608 1908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:11.887959 1908 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:41369
I20250813 07:53:11.888000 1975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:41369 every 8 connection(s)
I20250813 07:53:11.890563 1908 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:53:11.895103 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 1908
I20250813 07:53:11.895424 1976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:11.895484 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:53:11.920575 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe: Bootstrap starting.
I20250813 07:53:11.925781 1976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:11.927412 1976 log.cc:826] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:11.931970 1976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe: No bootstrap required, opened a new log
I20250813 07:53:11.948990 1976 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } }
I20250813 07:53:11.949735 1976 raft_consensus.cc:383] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:11.949975 1976 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98166fc4ec89470c91f605ff0e7eafbe, State: Initialized, Role: FOLLOWER
I20250813 07:53:11.950636 1976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } }
I20250813 07:53:11.951220 1976 raft_consensus.cc:397] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:11.951472 1976 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:11.951761 1976 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:11.956182 1976 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } }
I20250813 07:53:11.956904 1976 leader_election.cc:304] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98166fc4ec89470c91f605ff0e7eafbe; no voters:
I20250813 07:53:11.958527 1976 leader_election.cc:290] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:11.959285 1981 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:11.961324 1981 raft_consensus.cc:695] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [term 1 LEADER]: Becoming Leader. State: Replica: 98166fc4ec89470c91f605ff0e7eafbe, State: Running, Role: LEADER
I20250813 07:53:11.962059 1981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } }
I20250813 07:53:11.963402 1976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:53:11.969178 1983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98166fc4ec89470c91f605ff0e7eafbe. Latest consensus state: current_term: 1 leader_uuid: "98166fc4ec89470c91f605ff0e7eafbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } } }
I20250813 07:53:11.969347 1982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98166fc4ec89470c91f605ff0e7eafbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98166fc4ec89470c91f605ff0e7eafbe" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 41369 } } }
I20250813 07:53:11.969898 1982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:11.969898 1983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:11.976159 1989 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:53:11.987133 1989 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:53:12.004567 1989 catalog_manager.cc:1349] Generated new cluster ID: bc68603be8414686a258a1a38a039cad
I20250813 07:53:12.004837 1989 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:53:12.056025 1989 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:53:12.058048 1989 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:53:12.076493 1989 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 98166fc4ec89470c91f605ff0e7eafbe: Generated new TSK 0
I20250813 07:53:12.077394 1989 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:53:12.094898 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--builtin_ntp_servers=127.0.106.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 07:53:12.386912 2000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:12.387418 2000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:12.387893 2000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:12.419201 2000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:12.420027 2000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:53:12.454270 2000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:12.455600 2000 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:12.457214 2000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:12.469121 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:12.471235 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:13.865485 2008 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1389 milliseconds
W20250813 07:53:13.865701 2000 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.396s user 0.457s sys 0.932s
W20250813 07:53:13.866343 2000 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.397s user 0.457s sys 0.933s
W20250813 07:53:13.868476 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:13.869714 2000 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:13.871116 2000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:13.873879 2000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:13.875432 2000 hybrid_clock.cc:648] HybridClock initialized: now 1755071593875342 us; error 96 us; skew 500 ppm
I20250813 07:53:13.876489 2000 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:13.885681 2000 webserver.cc:480] Webserver started at http://127.0.106.193:35619/ using document root <none> and password file <none>
I20250813 07:53:13.887015 2000 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:13.887341 2000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:13.887974 2000 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:13.894518 2000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "069d54e8b0c940c3adc7bee6302eebf0"
format_stamp: "Formatted at 2025-08-13 07:53:13 on dist-test-slave-dr1z"
I20250813 07:53:13.896114 2000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "069d54e8b0c940c3adc7bee6302eebf0"
format_stamp: "Formatted at 2025-08-13 07:53:13 on dist-test-slave-dr1z"
I20250813 07:53:13.906224 2000 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20250813 07:53:13.914508 2016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:13.915966 2000 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250813 07:53:13.916381 2000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "069d54e8b0c940c3adc7bee6302eebf0"
format_stamp: "Formatted at 2025-08-13 07:53:13 on dist-test-slave-dr1z"
I20250813 07:53:13.916811 2000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:14.048645 2000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:14.050145 2000 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:14.050576 2000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:14.053365 2000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:14.057587 2000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:14.057793 2000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:14.058048 2000 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:14.058207 2000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:14.218166 2000 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:39315
I20250813 07:53:14.218274 2128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:39315 every 8 connection(s)
I20250813 07:53:14.220772 2000 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:53:14.227916 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2000
I20250813 07:53:14.228402 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:53:14.235889 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--builtin_ntp_servers=127.0.106.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:14.248740 2129 heartbeater.cc:344] Connected to a master server at 127.0.106.254:41369
I20250813 07:53:14.249176 2129 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:14.250238 2129 heartbeater.cc:507] Master 127.0.106.254:41369 requested a full tablet report, sending...
I20250813 07:53:14.252699 1941 ts_manager.cc:194] Registered new tserver with Master: 069d54e8b0c940c3adc7bee6302eebf0 (127.0.106.193:39315)
I20250813 07:53:14.254791 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:53173
W20250813 07:53:14.542517 2133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:14.542995 2133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:14.543499 2133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:14.574429 2133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:14.575273 2133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:53:14.610410 2133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:14.611721 2133 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:14.613293 2133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:14.624459 2139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:15.258131 2129 heartbeater.cc:499] Master 127.0.106.254:41369 was elected leader, sending a full tablet report...
W20250813 07:53:16.027812 2138 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2133
W20250813 07:53:16.140185 2133 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.515s user 0.635s sys 0.879s
W20250813 07:53:14.625622 2140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:16.140574 2133 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.515s user 0.635s sys 0.880s
W20250813 07:53:16.142702 2142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:16.145048 2141 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1515 milliseconds
I20250813 07:53:16.145113 2133 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:16.146250 2133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:16.148226 2133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:16.149600 2133 hybrid_clock.cc:648] HybridClock initialized: now 1755071596149537 us; error 61 us; skew 500 ppm
I20250813 07:53:16.150385 2133 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:16.156379 2133 webserver.cc:480] Webserver started at http://127.0.106.194:45639/ using document root <none> and password file <none>
I20250813 07:53:16.157279 2133 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:16.157493 2133 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:16.157938 2133 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:16.162256 2133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6bb2de403d59429cb5cb825500405e54"
format_stamp: "Formatted at 2025-08-13 07:53:16 on dist-test-slave-dr1z"
I20250813 07:53:16.163401 2133 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6bb2de403d59429cb5cb825500405e54"
format_stamp: "Formatted at 2025-08-13 07:53:16 on dist-test-slave-dr1z"
I20250813 07:53:16.170502 2133 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250813 07:53:16.175951 2149 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:16.177076 2133 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250813 07:53:16.177384 2133 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6bb2de403d59429cb5cb825500405e54"
format_stamp: "Formatted at 2025-08-13 07:53:16 on dist-test-slave-dr1z"
I20250813 07:53:16.177758 2133 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:16.247776 2133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:16.249161 2133 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:16.249595 2133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:16.251968 2133 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:16.255962 2133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:16.256152 2133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:16.256350 2133 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:16.256489 2133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:16.381940 2133 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:36095
I20250813 07:53:16.382052 2261 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:36095 every 8 connection(s)
I20250813 07:53:16.384554 2133 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:53:16.393790 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2133
I20250813 07:53:16.394282 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:53:16.400324 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--builtin_ntp_servers=127.0.106.212:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:16.405618 2262 heartbeater.cc:344] Connected to a master server at 127.0.106.254:41369
I20250813 07:53:16.406025 2262 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:16.406993 2262 heartbeater.cc:507] Master 127.0.106.254:41369 requested a full tablet report, sending...
I20250813 07:53:16.409083 1941 ts_manager.cc:194] Registered new tserver with Master: 6bb2de403d59429cb5cb825500405e54 (127.0.106.194:36095)
I20250813 07:53:16.410357 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:37859
W20250813 07:53:16.695075 2266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:16.695577 2266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:16.696092 2266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:16.727067 2266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:16.727937 2266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:53:16.762480 2266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:41369
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:16.763790 2266 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:16.765444 2266 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:16.776796 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:17.413764 2262 heartbeater.cc:499] Master 127.0.106.254:41369 was elected leader, sending a full tablet report...
W20250813 07:53:16.778287 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:17.884732 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:17.887101 2275 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250813 07:53:17.887174 2266 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:17.888301 2266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:17.890789 2266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:17.892201 2266 hybrid_clock.cc:648] HybridClock initialized: now 1755071597892142 us; error 64 us; skew 500 ppm
I20250813 07:53:17.893005 2266 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:17.899232 2266 webserver.cc:480] Webserver started at http://127.0.106.195:41661/ using document root <none> and password file <none>
I20250813 07:53:17.900147 2266 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:17.900331 2266 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:17.900779 2266 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:17.905086 2266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "af7fb4b981d54a8cbc07c4bb524c1341"
format_stamp: "Formatted at 2025-08-13 07:53:17 on dist-test-slave-dr1z"
I20250813 07:53:17.906177 2266 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "af7fb4b981d54a8cbc07c4bb524c1341"
format_stamp: "Formatted at 2025-08-13 07:53:17 on dist-test-slave-dr1z"
I20250813 07:53:17.913143 2266 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250813 07:53:17.918694 2283 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:17.919708 2266 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250813 07:53:17.920049 2266 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "af7fb4b981d54a8cbc07c4bb524c1341"
format_stamp: "Formatted at 2025-08-13 07:53:17 on dist-test-slave-dr1z"
I20250813 07:53:17.920360 2266 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:17.978917 2266 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:17.980544 2266 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:17.981060 2266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:17.983793 2266 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:17.987918 2266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:17.988131 2266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:17.988340 2266 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:17.988476 2266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:18.113665 2266 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:33389
I20250813 07:53:18.113775 2395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:33389 every 8 connection(s)
I20250813 07:53:18.116127 2266 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:53:18.117825 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2266
I20250813 07:53:18.118556 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:53:18.136430 2396 heartbeater.cc:344] Connected to a master server at 127.0.106.254:41369
I20250813 07:53:18.136940 2396 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:18.138084 2396 heartbeater.cc:507] Master 127.0.106.254:41369 requested a full tablet report, sending...
I20250813 07:53:18.140204 1941 ts_manager.cc:194] Registered new tserver with Master: af7fb4b981d54a8cbc07c4bb524c1341 (127.0.106.195:33389)
I20250813 07:53:18.141379 1941 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:47305
I20250813 07:53:18.153635 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:18.186116 1941 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44832:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 07:53:18.203980 1941 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:53:18.264261 2197 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5fa7f4d6449019577baa36f5bd6e6 (DEFAULT_TABLE table=TestTable [id=d0bb90bcfb3b4e19a97acb02fc05468e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:18.267952 2197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5fa7f4d6449019577baa36f5bd6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.267271 2331 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5fa7f4d6449019577baa36f5bd6e6 (DEFAULT_TABLE table=TestTable [id=d0bb90bcfb3b4e19a97acb02fc05468e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:18.269403 2331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5fa7f4d6449019577baa36f5bd6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.279078 2064 tablet_service.cc:1468] Processing CreateTablet for tablet 65e5fa7f4d6449019577baa36f5bd6e6 (DEFAULT_TABLE table=TestTable [id=d0bb90bcfb3b4e19a97acb02fc05468e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:18.280668 2064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e5fa7f4d6449019577baa36f5bd6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.296576 2415 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Bootstrap starting.
I20250813 07:53:18.299135 2416 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Bootstrap starting.
I20250813 07:53:18.302855 2417 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Bootstrap starting.
I20250813 07:53:18.305531 2415 tablet_bootstrap.cc:654] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.305733 2416 tablet_bootstrap.cc:654] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.307281 2415 log.cc:826] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:18.308041 2416 log.cc:826] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:18.310529 2417 tablet_bootstrap.cc:654] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.312299 2415 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: No bootstrap required, opened a new log
I20250813 07:53:18.312686 2415 ts_tablet_manager.cc:1397] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20250813 07:53:18.312629 2417 log.cc:826] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:18.314085 2416 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: No bootstrap required, opened a new log
I20250813 07:53:18.314710 2416 ts_tablet_manager.cc:1397] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250813 07:53:18.317977 2417 tablet_bootstrap.cc:492] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: No bootstrap required, opened a new log
I20250813 07:53:18.318512 2417 ts_tablet_manager.cc:1397] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20250813 07:53:18.334124 2415 raft_consensus.cc:357] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.335328 2415 raft_consensus.cc:383] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.335680 2415 raft_consensus.cc:738] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7fb4b981d54a8cbc07c4bb524c1341, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.336733 2415 consensus_queue.cc:260] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.340720 2416 raft_consensus.cc:357] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.341737 2416 raft_consensus.cc:383] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.342092 2416 raft_consensus.cc:738] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb2de403d59429cb5cb825500405e54, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.343034 2416 consensus_queue.cc:260] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.344872 2417 raft_consensus.cc:357] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.345935 2417 raft_consensus.cc:383] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.346279 2417 raft_consensus.cc:738] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069d54e8b0c940c3adc7bee6302eebf0, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.347923 2415 ts_tablet_manager.cc:1428] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Time spent starting tablet: real 0.035s user 0.028s sys 0.004s
I20250813 07:53:18.347364 2417 consensus_queue.cc:260] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.348771 2396 heartbeater.cc:499] Master 127.0.106.254:41369 was elected leader, sending a full tablet report...
I20250813 07:53:18.350364 2416 ts_tablet_manager.cc:1428] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Time spent starting tablet: real 0.035s user 0.029s sys 0.006s
I20250813 07:53:18.353200 2417 ts_tablet_manager.cc:1428] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Time spent starting tablet: real 0.034s user 0.029s sys 0.004s
W20250813 07:53:18.370553 2397 tablet.cc:2378] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:53:18.390889 2263 tablet.cc:2378] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:53:18.485280 2130 tablet.cc:2378] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:18.634886 2422 raft_consensus.cc:491] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:53:18.635393 2422 raft_consensus.cc:513] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.637936 2422 leader_election.cc:290] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af7fb4b981d54a8cbc07c4bb524c1341 (127.0.106.195:33389), 069d54e8b0c940c3adc7bee6302eebf0 (127.0.106.193:39315)
I20250813 07:53:18.649237 2351 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5fa7f4d6449019577baa36f5bd6e6" candidate_uuid: "6bb2de403d59429cb5cb825500405e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" is_pre_election: true
I20250813 07:53:18.650072 2351 raft_consensus.cc:2466] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb2de403d59429cb5cb825500405e54 in term 0.
I20250813 07:53:18.650349 2084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5fa7f4d6449019577baa36f5bd6e6" candidate_uuid: "6bb2de403d59429cb5cb825500405e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d54e8b0c940c3adc7bee6302eebf0" is_pre_election: true
I20250813 07:53:18.651113 2084 raft_consensus.cc:2466] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb2de403d59429cb5cb825500405e54 in term 0.
I20250813 07:53:18.651135 2151 leader_election.cc:304] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bb2de403d59429cb5cb825500405e54, af7fb4b981d54a8cbc07c4bb524c1341; no voters:
I20250813 07:53:18.651830 2422 raft_consensus.cc:2802] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:53:18.652172 2422 raft_consensus.cc:491] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:53:18.652488 2422 raft_consensus.cc:3058] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.656651 2422 raft_consensus.cc:513] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.658064 2422 leader_election.cc:290] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [CANDIDATE]: Term 1 election: Requested vote from peers af7fb4b981d54a8cbc07c4bb524c1341 (127.0.106.195:33389), 069d54e8b0c940c3adc7bee6302eebf0 (127.0.106.193:39315)
I20250813 07:53:18.658758 2351 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5fa7f4d6449019577baa36f5bd6e6" candidate_uuid: "6bb2de403d59429cb5cb825500405e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af7fb4b981d54a8cbc07c4bb524c1341"
I20250813 07:53:18.659090 2084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65e5fa7f4d6449019577baa36f5bd6e6" candidate_uuid: "6bb2de403d59429cb5cb825500405e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d54e8b0c940c3adc7bee6302eebf0"
I20250813 07:53:18.659294 2351 raft_consensus.cc:3058] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.659600 2084 raft_consensus.cc:3058] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.666054 2351 raft_consensus.cc:2466] T 65e5fa7f4d6449019577baa36f5bd6e6 P af7fb4b981d54a8cbc07c4bb524c1341 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb2de403d59429cb5cb825500405e54 in term 1.
I20250813 07:53:18.666049 2084 raft_consensus.cc:2466] T 65e5fa7f4d6449019577baa36f5bd6e6 P 069d54e8b0c940c3adc7bee6302eebf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb2de403d59429cb5cb825500405e54 in term 1.
I20250813 07:53:18.667039 2151 leader_election.cc:304] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bb2de403d59429cb5cb825500405e54, af7fb4b981d54a8cbc07c4bb524c1341; no voters:
I20250813 07:53:18.667670 2422 raft_consensus.cc:2802] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:18.669200 2422 raft_consensus.cc:695] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb2de403d59429cb5cb825500405e54, State: Running, Role: LEADER
I20250813 07:53:18.669950 2422 consensus_queue.cc:237] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } }
I20250813 07:53:18.680002 1939 catalog_manager.cc:5582] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bb2de403d59429cb5cb825500405e54 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "6bb2de403d59429cb5cb825500405e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } health_report { overall_health: UNKNOWN } } }
I20250813 07:53:18.732928 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:18.737046 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 069d54e8b0c940c3adc7bee6302eebf0 to finish bootstrapping
I20250813 07:53:18.753387 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6bb2de403d59429cb5cb825500405e54 to finish bootstrapping
I20250813 07:53:18.766371 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver af7fb4b981d54a8cbc07c4bb524c1341 to finish bootstrapping
I20250813 07:53:18.781177 1939 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44832:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250813 07:53:18.782678 1939 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:53:18.798687 2064 tablet_service.cc:1468] Processing CreateTablet for tablet 2faf370229ff4190b5ae2f27f3796884 (DEFAULT_TABLE table=TestAnotherTable [id=6dde232c3b234b258e86fdcd92fd1d3e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 07:53:18.799248 2197 tablet_service.cc:1468] Processing CreateTablet for tablet 2faf370229ff4190b5ae2f27f3796884 (DEFAULT_TABLE table=TestAnotherTable [id=6dde232c3b234b258e86fdcd92fd1d3e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 07:53:18.799779 2064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2faf370229ff4190b5ae2f27f3796884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.799911 2331 tablet_service.cc:1468] Processing CreateTablet for tablet 2faf370229ff4190b5ae2f27f3796884 (DEFAULT_TABLE table=TestAnotherTable [id=6dde232c3b234b258e86fdcd92fd1d3e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 07:53:18.800266 2197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2faf370229ff4190b5ae2f27f3796884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.800760 2331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2faf370229ff4190b5ae2f27f3796884. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:18.811960 2415 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341: Bootstrap starting.
I20250813 07:53:18.813421 2417 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0: Bootstrap starting.
I20250813 07:53:18.815198 2416 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54: Bootstrap starting.
I20250813 07:53:18.817700 2415 tablet_bootstrap.cc:654] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.818650 2417 tablet_bootstrap.cc:654] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.821204 2416 tablet_bootstrap.cc:654] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:18.825976 2415 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341: No bootstrap required, opened a new log
I20250813 07:53:18.826438 2415 ts_tablet_manager.cc:1397] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20250813 07:53:18.827386 2416 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54: No bootstrap required, opened a new log
I20250813 07:53:18.827724 2416 ts_tablet_manager.cc:1397] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20250813 07:53:18.828452 2415 raft_consensus.cc:357] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.829195 2415 raft_consensus.cc:383] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.829432 2417 tablet_bootstrap.cc:492] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0: No bootstrap required, opened a new log
I20250813 07:53:18.829530 2415 raft_consensus.cc:738] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7fb4b981d54a8cbc07c4bb524c1341, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.829916 2417 ts_tablet_manager.cc:1397] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20250813 07:53:18.830219 2416 raft_consensus.cc:357] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.830389 2415 consensus_queue.cc:260] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.830950 2416 raft_consensus.cc:383] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.831274 2416 raft_consensus.cc:738] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb2de403d59429cb5cb825500405e54, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.832382 2415 ts_tablet_manager.cc:1428] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250813 07:53:18.832017 2416 consensus_queue.cc:260] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.832564 2417 raft_consensus.cc:357] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.833213 2417 raft_consensus.cc:383] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:18.833535 2417 raft_consensus.cc:738] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069d54e8b0c940c3adc7bee6302eebf0, State: Initialized, Role: FOLLOWER
I20250813 07:53:18.834183 2417 consensus_queue.cc:260] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.835739 2416 ts_tablet_manager.cc:1428] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250813 07:53:18.836104 2421 raft_consensus.cc:491] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:53:18.836541 2421 raft_consensus.cc:513] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.837512 2417 ts_tablet_manager.cc:1428] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20250813 07:53:18.843681 2421 leader_election.cc:290] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 069d54e8b0c940c3adc7bee6302eebf0 (127.0.106.193:39315), 6bb2de403d59429cb5cb825500405e54 (127.0.106.194:36095)
I20250813 07:53:18.857020 2084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2faf370229ff4190b5ae2f27f3796884" candidate_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d54e8b0c940c3adc7bee6302eebf0" is_pre_election: true
I20250813 07:53:18.857542 2084 raft_consensus.cc:2466] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7fb4b981d54a8cbc07c4bb524c1341 in term 0.
I20250813 07:53:18.858363 2217 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2faf370229ff4190b5ae2f27f3796884" candidate_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb2de403d59429cb5cb825500405e54" is_pre_election: true
I20250813 07:53:18.858500 2285 leader_election.cc:304] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069d54e8b0c940c3adc7bee6302eebf0, af7fb4b981d54a8cbc07c4bb524c1341; no voters:
I20250813 07:53:18.858965 2217 raft_consensus.cc:2466] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7fb4b981d54a8cbc07c4bb524c1341 in term 0.
I20250813 07:53:18.859192 2421 raft_consensus.cc:2802] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:53:18.859464 2421 raft_consensus.cc:491] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:53:18.859719 2421 raft_consensus.cc:3058] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.864148 2421 raft_consensus.cc:513] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.865783 2421 leader_election.cc:290] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [CANDIDATE]: Term 1 election: Requested vote from peers 069d54e8b0c940c3adc7bee6302eebf0 (127.0.106.193:39315), 6bb2de403d59429cb5cb825500405e54 (127.0.106.194:36095)
I20250813 07:53:18.866446 2084 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2faf370229ff4190b5ae2f27f3796884" candidate_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d54e8b0c940c3adc7bee6302eebf0"
I20250813 07:53:18.866678 2217 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2faf370229ff4190b5ae2f27f3796884" candidate_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bb2de403d59429cb5cb825500405e54"
I20250813 07:53:18.866969 2084 raft_consensus.cc:3058] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.867177 2217 raft_consensus.cc:3058] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:18.873064 2084 raft_consensus.cc:2466] T 2faf370229ff4190b5ae2f27f3796884 P 069d54e8b0c940c3adc7bee6302eebf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7fb4b981d54a8cbc07c4bb524c1341 in term 1.
I20250813 07:53:18.873384 2217 raft_consensus.cc:2466] T 2faf370229ff4190b5ae2f27f3796884 P 6bb2de403d59429cb5cb825500405e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7fb4b981d54a8cbc07c4bb524c1341 in term 1.
I20250813 07:53:18.874055 2285 leader_election.cc:304] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069d54e8b0c940c3adc7bee6302eebf0, af7fb4b981d54a8cbc07c4bb524c1341; no voters:
I20250813 07:53:18.874780 2421 raft_consensus.cc:2802] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:18.876657 2421 raft_consensus.cc:695] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [term 1 LEADER]: Becoming Leader. State: Replica: af7fb4b981d54a8cbc07c4bb524c1341, State: Running, Role: LEADER
I20250813 07:53:18.877612 2421 consensus_queue.cc:237] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } }
I20250813 07:53:18.887936 1941 catalog_manager.cc:5582] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 reported cstate change: term changed from 0 to 1, leader changed from <none> to af7fb4b981d54a8cbc07c4bb524c1341 (127.0.106.195). New cstate: current_term: 1 leader_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:19.097716 2422 consensus_queue.cc:1035] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af7fb4b981d54a8cbc07c4bb524c1341" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:53:19.132299 2422 consensus_queue.cc:1035] T 65e5fa7f4d6449019577baa36f5bd6e6 P 6bb2de403d59429cb5cb825500405e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 07:53:19.257282 2436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:19.258095 2436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:19.297204 2436 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250813 07:53:19.356868 2421 consensus_queue.cc:1035] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "069d54e8b0c940c3adc7bee6302eebf0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 39315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:53:19.371354 2434 consensus_queue.cc:1035] T 2faf370229ff4190b5ae2f27f3796884 P af7fb4b981d54a8cbc07c4bb524c1341 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bb2de403d59429cb5cb825500405e54" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 07:53:20.534404 2436 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.201s user 0.461s sys 0.729s
W20250813 07:53:20.534780 2436 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.202s user 0.461s sys 0.729s
W20250813 07:53:21.903832 2466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:21.904502 2466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:21.935300 2466 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 07:53:23.118345 2466 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.146s user 0.458s sys 0.686s
W20250813 07:53:23.118736 2466 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.146s user 0.458s sys 0.686s
W20250813 07:53:24.412796 1972 debug-util.cc:398] Leaking SignalData structure 0x7b080006faa0 after lost signal to thread 1909
W20250813 07:53:24.413556 1972 debug-util.cc:398] Leaking SignalData structure 0x7b08000a88a0 after lost signal to thread 1975
W20250813 07:53:24.523196 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:24.523797 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:24.555290 2480 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 07:53:25.858162 2480 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.263s user 0.492s sys 0.765s
W20250813 07:53:25.858544 2480 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.263s user 0.492s sys 0.765s
W20250813 07:53:27.255815 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:27.256431 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:27.289551 2496 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 07:53:28.485522 2496 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.159s user 0.526s sys 0.630s
W20250813 07:53:28.485903 2496 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.159s user 0.529s sys 0.630s
I20250813 07:53:29.557231 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2000
I20250813 07:53:29.578230 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2133
I20250813 07:53:29.599562 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2266
I20250813 07:53:29.621172 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 1908
2025-08-13T07:53:29Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19500 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250813 07:53:29.677063 427 test_util.cc:276] Using random seed: 1943643372
I20250813 07:53:29.681089 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:53:29.681278 427 ts_itest-base.cc:116] --------------
I20250813 07:53:29.681427 427 ts_itest-base.cc:117] 3 tablet servers
I20250813 07:53:29.681591 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:53:29.681731 427 ts_itest-base.cc:119] --------------
2025-08-13T07:53:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:53:29Z Disabled control of system clock
I20250813 07:53:29.715312 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:44847
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:44847
--superuser_acl=no-such-user with env {}
W20250813 07:53:30.004500 2516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:30.005053 2516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:30.005523 2516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:30.037269 2516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:53:30.037616 2516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:30.037856 2516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:53:30.038077 2516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:53:30.073376 2516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:44847
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:30.074710 2516 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:30.076295 2516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:30.086930 2522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:30.087260 2523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:31.203186 2525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:31.204286 2524 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1113 milliseconds
W20250813 07:53:31.205138 2516 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.119s user 0.424s sys 0.683s
W20250813 07:53:31.205385 2516 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.119s user 0.424s sys 0.683s
I20250813 07:53:31.205654 2516 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:31.206694 2516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:31.209631 2516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:31.211005 2516 hybrid_clock.cc:648] HybridClock initialized: now 1755071611210955 us; error 59 us; skew 500 ppm
I20250813 07:53:31.211812 2516 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:31.218978 2516 webserver.cc:480] Webserver started at http://127.0.106.254:44343/ using document root <none> and password file <none>
I20250813 07:53:31.219928 2516 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:31.220155 2516 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:31.220597 2516 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:31.225077 2516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "179c058f588c4d73b2c43132254850d5"
format_stamp: "Formatted at 2025-08-13 07:53:31 on dist-test-slave-dr1z"
I20250813 07:53:31.226227 2516 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "179c058f588c4d73b2c43132254850d5"
format_stamp: "Formatted at 2025-08-13 07:53:31 on dist-test-slave-dr1z"
I20250813 07:53:31.234205 2516 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250813 07:53:31.240134 2532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:31.241328 2516 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 07:53:31.241734 2516 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "179c058f588c4d73b2c43132254850d5"
format_stamp: "Formatted at 2025-08-13 07:53:31 on dist-test-slave-dr1z"
I20250813 07:53:31.242082 2516 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:31.316485 2516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:31.318032 2516 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:31.318473 2516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:31.385789 2516 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:44847
I20250813 07:53:31.385859 2583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:44847 every 8 connection(s)
I20250813 07:53:31.388402 2516 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:53:31.393590 2584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:31.393733 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2516
I20250813 07:53:31.394107 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:53:31.416049 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5: Bootstrap starting.
I20250813 07:53:31.422025 2584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:31.423727 2584 log.cc:826] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:31.428274 2584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5: No bootstrap required, opened a new log
I20250813 07:53:31.445354 2584 raft_consensus.cc:357] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } }
I20250813 07:53:31.446074 2584 raft_consensus.cc:383] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:31.446302 2584 raft_consensus.cc:738] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179c058f588c4d73b2c43132254850d5, State: Initialized, Role: FOLLOWER
I20250813 07:53:31.447029 2584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } }
I20250813 07:53:31.447629 2584 raft_consensus.cc:397] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:31.447921 2584 raft_consensus.cc:491] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:31.448217 2584 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:31.452158 2584 raft_consensus.cc:513] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } }
I20250813 07:53:31.452828 2584 leader_election.cc:304] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179c058f588c4d73b2c43132254850d5; no voters:
I20250813 07:53:31.454420 2584 leader_election.cc:290] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:31.455149 2589 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:31.457203 2589 raft_consensus.cc:695] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [term 1 LEADER]: Becoming Leader. State: Replica: 179c058f588c4d73b2c43132254850d5, State: Running, Role: LEADER
I20250813 07:53:31.458418 2584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:53:31.458158 2589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } }
I20250813 07:53:31.466147 2590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179c058f588c4d73b2c43132254850d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } } }
I20250813 07:53:31.466984 2590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:31.466742 2591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179c058f588c4d73b2c43132254850d5. Latest consensus state: current_term: 1 leader_uuid: "179c058f588c4d73b2c43132254850d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179c058f588c4d73b2c43132254850d5" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 44847 } } }
I20250813 07:53:31.467460 2591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:31.473778 2597 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:53:31.485025 2597 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:53:31.500862 2597 catalog_manager.cc:1349] Generated new cluster ID: 47aa3cb2e5c84bb9931d03de6b3ebb18
I20250813 07:53:31.501178 2597 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:53:31.528625 2597 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:53:31.530085 2597 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:53:31.547266 2597 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 179c058f588c4d73b2c43132254850d5: Generated new TSK 0
I20250813 07:53:31.548151 2597 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:53:31.559818 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--builtin_ntp_servers=127.0.106.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 07:53:31.854139 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:31.854725 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:31.855203 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:31.890003 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:31.890944 2608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:53:31.925666 2608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:31.927711 2608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:31.930106 2608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:31.942955 2614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:31.946962 2617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:31.947662 2608 server_base.cc:1047] running on GCE node
W20250813 07:53:31.947283 2615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:33.045046 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:33.047724 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:33.049166 2608 hybrid_clock.cc:648] HybridClock initialized: now 1755071613049100 us; error 76 us; skew 500 ppm
I20250813 07:53:33.050014 2608 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:33.056375 2608 webserver.cc:480] Webserver started at http://127.0.106.193:42929/ using document root <none> and password file <none>
I20250813 07:53:33.057487 2608 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:33.057919 2608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:33.058437 2608 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:33.063014 2608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c377ff17dfa8404ab69dc2eb33372444"
format_stamp: "Formatted at 2025-08-13 07:53:33 on dist-test-slave-dr1z"
I20250813 07:53:33.064208 2608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c377ff17dfa8404ab69dc2eb33372444"
format_stamp: "Formatted at 2025-08-13 07:53:33 on dist-test-slave-dr1z"
I20250813 07:53:33.071444 2608 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250813 07:53:33.077757 2624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:33.078967 2608 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:53:33.079288 2608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c377ff17dfa8404ab69dc2eb33372444"
format_stamp: "Formatted at 2025-08-13 07:53:33 on dist-test-slave-dr1z"
I20250813 07:53:33.079629 2608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:33.137879 2608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:33.139314 2608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:33.139760 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:33.142577 2608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:33.146936 2608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:33.147164 2608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:33.147398 2608 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:33.147549 2608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:33.279033 2608 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:43147
I20250813 07:53:33.279104 2736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:43147 every 8 connection(s)
I20250813 07:53:33.282392 2608 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:53:33.286912 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2608
I20250813 07:53:33.287338 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:53:33.293979 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--builtin_ntp_servers=127.0.106.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:33.306095 2737 heartbeater.cc:344] Connected to a master server at 127.0.106.254:44847
I20250813 07:53:33.306526 2737 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:33.307548 2737 heartbeater.cc:507] Master 127.0.106.254:44847 requested a full tablet report, sending...
I20250813 07:53:33.310252 2549 ts_manager.cc:194] Registered new tserver with Master: c377ff17dfa8404ab69dc2eb33372444 (127.0.106.193:43147)
I20250813 07:53:33.312965 2549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:53193
W20250813 07:53:33.587958 2741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:33.588410 2741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:33.588863 2741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:33.619951 2741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:33.620815 2741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:53:33.654646 2741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:33.655859 2741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:33.657429 2741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:33.668671 2747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:34.316571 2737 heartbeater.cc:499] Master 127.0.106.254:44847 was elected leader, sending a full tablet report...
W20250813 07:53:33.670122 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:34.861719 2750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:34.865445 2749 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1194 milliseconds
W20250813 07:53:34.865675 2741 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.197s user 0.462s sys 0.723s
W20250813 07:53:34.866014 2741 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.197s user 0.462s sys 0.723s
I20250813 07:53:34.866212 2741 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:34.867208 2741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:34.869354 2741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:34.870740 2741 hybrid_clock.cc:648] HybridClock initialized: now 1755071614870715 us; error 45 us; skew 500 ppm
I20250813 07:53:34.871510 2741 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:34.878755 2741 webserver.cc:480] Webserver started at http://127.0.106.194:44589/ using document root <none> and password file <none>
I20250813 07:53:34.879757 2741 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:34.879979 2741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:34.880420 2741 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:34.884841 2741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c618c10580f5482e88f8f3a5932c995c"
format_stamp: "Formatted at 2025-08-13 07:53:34 on dist-test-slave-dr1z"
I20250813 07:53:34.885982 2741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c618c10580f5482e88f8f3a5932c995c"
format_stamp: "Formatted at 2025-08-13 07:53:34 on dist-test-slave-dr1z"
I20250813 07:53:34.893632 2741 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20250813 07:53:34.899497 2757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:34.900565 2741 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250813 07:53:34.900876 2741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c618c10580f5482e88f8f3a5932c995c"
format_stamp: "Formatted at 2025-08-13 07:53:34 on dist-test-slave-dr1z"
I20250813 07:53:34.901204 2741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:34.965369 2741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:34.966854 2741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:34.967276 2741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:34.969739 2741 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:34.973716 2741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:34.973937 2741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:34.974175 2741 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:34.974329 2741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:35.103046 2741 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:35063
I20250813 07:53:35.103183 2869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:35063 every 8 connection(s)
I20250813 07:53:35.105599 2741 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:53:35.106827 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2741
I20250813 07:53:35.107352 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:53:35.114827 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--builtin_ntp_servers=127.0.106.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:35.128949 2870 heartbeater.cc:344] Connected to a master server at 127.0.106.254:44847
I20250813 07:53:35.129385 2870 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:35.130410 2870 heartbeater.cc:507] Master 127.0.106.254:44847 requested a full tablet report, sending...
I20250813 07:53:35.132583 2549 ts_manager.cc:194] Registered new tserver with Master: c618c10580f5482e88f8f3a5932c995c (127.0.106.194:35063)
I20250813 07:53:35.133802 2549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:44317
W20250813 07:53:35.409528 2874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:35.410014 2874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:35.410482 2874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:35.441197 2874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:35.442063 2874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
W20250813 07:53:35.474921 2580 debug-util.cc:398] Leaking SignalData structure 0x7b080006fee0 after lost signal to thread 2517
I20250813 07:53:35.476387 2874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:44847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:35.477694 2874 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:35.479499 2874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:35.491560 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:36.137029 2870 heartbeater.cc:499] Master 127.0.106.254:44847 was elected leader, sending a full tablet report...
W20250813 07:53:35.495640 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:35.493995 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:36.696300 2882 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1199 milliseconds
I20250813 07:53:36.696410 2874 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:36.697779 2874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:36.700189 2874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:36.701570 2874 hybrid_clock.cc:648] HybridClock initialized: now 1755071616701518 us; error 71 us; skew 500 ppm
I20250813 07:53:36.702335 2874 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:36.708664 2874 webserver.cc:480] Webserver started at http://127.0.106.195:46209/ using document root <none> and password file <none>
I20250813 07:53:36.709578 2874 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:36.709782 2874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:36.710219 2874 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:36.714665 2874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c94472d209a944bda88c96b96896b46f"
format_stamp: "Formatted at 2025-08-13 07:53:36 on dist-test-slave-dr1z"
I20250813 07:53:36.715724 2874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c94472d209a944bda88c96b96896b46f"
format_stamp: "Formatted at 2025-08-13 07:53:36 on dist-test-slave-dr1z"
I20250813 07:53:36.723105 2874 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250813 07:53:36.729081 2891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:36.730149 2874 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:53:36.730454 2874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c94472d209a944bda88c96b96896b46f"
format_stamp: "Formatted at 2025-08-13 07:53:36 on dist-test-slave-dr1z"
I20250813 07:53:36.730760 2874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:36.796764 2874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:36.798270 2874 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:36.798688 2874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:36.801643 2874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:36.805938 2874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:36.806195 2874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:36.806459 2874 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:36.806604 2874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:36.959527 2874 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:35447
I20250813 07:53:36.959713 3003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:35447 every 8 connection(s)
I20250813 07:53:36.962210 2874 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:53:36.968462 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 2874
I20250813 07:53:36.969138 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:53:36.997880 3004 heartbeater.cc:344] Connected to a master server at 127.0.106.254:44847
I20250813 07:53:36.998432 3004 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:36.999756 3004 heartbeater.cc:507] Master 127.0.106.254:44847 requested a full tablet report, sending...
I20250813 07:53:37.002380 2549 ts_manager.cc:194] Registered new tserver with Master: c94472d209a944bda88c96b96896b46f (127.0.106.195:35447)
I20250813 07:53:37.003835 2549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:55379
I20250813 07:53:37.005765 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:37.046471 2548 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58850:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 07:53:37.066206 2548 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:53:37.118768 2939 tablet_service.cc:1468] Processing CreateTablet for tablet 46c6ae9c3aa545ccb7d52c28af600c62 (DEFAULT_TABLE table=TestTable [id=a185a4b06b1d4fed9d32089b599e766f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:37.121387 2939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c6ae9c3aa545ccb7d52c28af600c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:37.154832 2805 tablet_service.cc:1468] Processing CreateTablet for tablet 46c6ae9c3aa545ccb7d52c28af600c62 (DEFAULT_TABLE table=TestTable [id=a185a4b06b1d4fed9d32089b599e766f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:37.156723 2805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c6ae9c3aa545ccb7d52c28af600c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:37.161911 3023 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Bootstrap starting.
I20250813 07:53:37.161566 2672 tablet_service.cc:1468] Processing CreateTablet for tablet 46c6ae9c3aa545ccb7d52c28af600c62 (DEFAULT_TABLE table=TestTable [id=a185a4b06b1d4fed9d32089b599e766f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:37.163439 2672 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46c6ae9c3aa545ccb7d52c28af600c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:37.168008 3023 tablet_bootstrap.cc:654] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:37.170413 3023 log.cc:826] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:37.193161 3023 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: No bootstrap required, opened a new log
I20250813 07:53:37.193867 3023 ts_tablet_manager.cc:1397] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20250813 07:53:37.210711 3025 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Bootstrap starting.
I20250813 07:53:37.211962 3026 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Bootstrap starting.
I20250813 07:53:37.221163 3026 tablet_bootstrap.cc:654] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:37.221163 3025 tablet_bootstrap.cc:654] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:37.221810 3023 raft_consensus.cc:357] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.222818 3023 raft_consensus.cc:383] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:37.223136 3023 raft_consensus.cc:738] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94472d209a944bda88c96b96896b46f, State: Initialized, Role: FOLLOWER
I20250813 07:53:37.223520 3025 log.cc:826] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:37.223520 3026 log.cc:826] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:37.224256 3023 consensus_queue.cc:260] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.229692 3023 ts_tablet_manager.cc:1428] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Time spent starting tablet: real 0.035s user 0.029s sys 0.002s
I20250813 07:53:37.230356 3004 heartbeater.cc:499] Master 127.0.106.254:44847 was elected leader, sending a full tablet report...
I20250813 07:53:37.246956 3025 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: No bootstrap required, opened a new log
I20250813 07:53:37.247409 3025 ts_tablet_manager.cc:1397] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.016s
I20250813 07:53:37.247807 3026 tablet_bootstrap.cc:492] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: No bootstrap required, opened a new log
I20250813 07:53:37.248306 3026 ts_tablet_manager.cc:1397] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.018s
I20250813 07:53:37.265904 3025 raft_consensus.cc:357] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.266603 3025 raft_consensus.cc:383] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:37.266839 3025 raft_consensus.cc:738] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c618c10580f5482e88f8f3a5932c995c, State: Initialized, Role: FOLLOWER
I20250813 07:53:37.267638 3029 raft_consensus.cc:491] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:53:37.267652 3025 consensus_queue.cc:260] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.268209 3029 raft_consensus.cc:513] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.273964 3026 raft_consensus.cc:357] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.274883 3026 raft_consensus.cc:383] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:37.275163 3026 raft_consensus.cc:738] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c377ff17dfa8404ab69dc2eb33372444, State: Initialized, Role: FOLLOWER
I20250813 07:53:37.276003 3026 consensus_queue.cc:260] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.285446 3029 leader_election.cc:290] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c377ff17dfa8404ab69dc2eb33372444 (127.0.106.193:43147), c618c10580f5482e88f8f3a5932c995c (127.0.106.194:35063)
I20250813 07:53:37.285615 2692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46c6ae9c3aa545ccb7d52c28af600c62" candidate_uuid: "c94472d209a944bda88c96b96896b46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c377ff17dfa8404ab69dc2eb33372444" is_pre_election: true
I20250813 07:53:37.286464 2692 raft_consensus.cc:2466] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94472d209a944bda88c96b96896b46f in term 0.
I20250813 07:53:37.287851 2892 leader_election.cc:304] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c377ff17dfa8404ab69dc2eb33372444, c94472d209a944bda88c96b96896b46f; no voters:
I20250813 07:53:37.288923 3029 raft_consensus.cc:2802] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:53:37.290082 3029 raft_consensus.cc:491] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:53:37.294031 3029 raft_consensus.cc:3058] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:37.316386 3029 raft_consensus.cc:513] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.324357 2825 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46c6ae9c3aa545ccb7d52c28af600c62" candidate_uuid: "c94472d209a944bda88c96b96896b46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c618c10580f5482e88f8f3a5932c995c" is_pre_election: true
I20250813 07:53:37.325271 2825 raft_consensus.cc:2466] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94472d209a944bda88c96b96896b46f in term 0.
I20250813 07:53:37.326421 2692 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46c6ae9c3aa545ccb7d52c28af600c62" candidate_uuid: "c94472d209a944bda88c96b96896b46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c377ff17dfa8404ab69dc2eb33372444"
I20250813 07:53:37.327001 2692 raft_consensus.cc:3058] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:37.340597 2692 raft_consensus.cc:2466] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94472d209a944bda88c96b96896b46f in term 1.
I20250813 07:53:37.342063 2892 leader_election.cc:304] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c377ff17dfa8404ab69dc2eb33372444, c94472d209a944bda88c96b96896b46f; no voters:
I20250813 07:53:37.350988 2825 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "46c6ae9c3aa545ccb7d52c28af600c62" candidate_uuid: "c94472d209a944bda88c96b96896b46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c618c10580f5482e88f8f3a5932c995c"
I20250813 07:53:37.351593 2825 raft_consensus.cc:3058] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:37.362102 2825 raft_consensus.cc:2466] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94472d209a944bda88c96b96896b46f in term 1.
I20250813 07:53:37.363938 3029 leader_election.cc:290] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [CANDIDATE]: Term 1 election: Requested vote from peers c377ff17dfa8404ab69dc2eb33372444 (127.0.106.193:43147), c618c10580f5482e88f8f3a5932c995c (127.0.106.194:35063)
I20250813 07:53:37.370030 3029 raft_consensus.cc:2802] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:37.371255 3029 raft_consensus.cc:695] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [term 1 LEADER]: Becoming Leader. State: Replica: c94472d209a944bda88c96b96896b46f, State: Running, Role: LEADER
I20250813 07:53:37.372752 3029 consensus_queue.cc:237] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } }
I20250813 07:53:37.379011 3025 ts_tablet_manager.cc:1428] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Time spent starting tablet: real 0.131s user 0.056s sys 0.056s
I20250813 07:53:37.379447 3026 ts_tablet_manager.cc:1428] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Time spent starting tablet: real 0.131s user 0.052s sys 0.068s
I20250813 07:53:37.397852 2548 catalog_manager.cc:5582] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f reported cstate change: term changed from 0 to 1, leader changed from <none> to c94472d209a944bda88c96b96896b46f (127.0.106.195). New cstate: current_term: 1 leader_uuid: "c94472d209a944bda88c96b96896b46f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94472d209a944bda88c96b96896b46f" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 35447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 } health_report { overall_health: UNKNOWN } } }
W20250813 07:53:37.467299 3005 tablet.cc:2378] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:37.468765 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:37.471910 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c377ff17dfa8404ab69dc2eb33372444 to finish bootstrapping
I20250813 07:53:37.485065 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c618c10580f5482e88f8f3a5932c995c to finish bootstrapping
I20250813 07:53:37.496856 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c94472d209a944bda88c96b96896b46f to finish bootstrapping
W20250813 07:53:37.540735 2738 tablet.cc:2378] T 46c6ae9c3aa545ccb7d52c28af600c62 P c377ff17dfa8404ab69dc2eb33372444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:53:37.620697 2871 tablet.cc:2378] T 46c6ae9c3aa545ccb7d52c28af600c62 P c618c10580f5482e88f8f3a5932c995c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:53:37.909974 3046 consensus_queue.cc:1035] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c377ff17dfa8404ab69dc2eb33372444" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 43147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250813 07:53:37.945372 3049 consensus_queue.cc:1035] T 46c6ae9c3aa545ccb7d52c28af600c62 P c94472d209a944bda88c96b96896b46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c618c10580f5482e88f8f3a5932c995c" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 35063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 07:53:39.073326 2548 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:58870
I20250813 07:53:40.100404 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2608
I20250813 07:53:40.133188 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2741
I20250813 07:53:40.154006 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2874
I20250813 07:53:40.175300 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 2516
2025-08-13T07:53:40Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10548 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250813 07:53:40.225214 427 test_util.cc:276] Using random seed: 1954191522
I20250813 07:53:40.229266 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:53:40.229453 427 ts_itest-base.cc:116] --------------
I20250813 07:53:40.229662 427 ts_itest-base.cc:117] 3 tablet servers
I20250813 07:53:40.229806 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:53:40.229939 427 ts_itest-base.cc:119] --------------
2025-08-13T07:53:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:53:40Z Disabled control of system clock
I20250813 07:53:40.263429 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:37727 with env {}
W20250813 07:53:40.556604 3074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:40.557193 3074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:40.557695 3074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:40.588563 3074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:53:40.588919 3074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:40.589170 3074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:53:40.589406 3074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:53:40.625905 3074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:37727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:40.627286 3074 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:40.628880 3074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:40.639324 3080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:40.639406 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:41.751499 3082 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1109 milliseconds
W20250813 07:53:41.752267 3083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:41.754161 3074 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.115s user 0.392s sys 0.718s
W20250813 07:53:41.754441 3074 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.115s user 0.392s sys 0.718s
I20250813 07:53:41.754655 3074 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:41.755709 3074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:41.758371 3074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:41.759764 3074 hybrid_clock.cc:648] HybridClock initialized: now 1755071621759751 us; error 87 us; skew 500 ppm
I20250813 07:53:41.760591 3074 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:41.768095 3074 webserver.cc:480] Webserver started at http://127.0.106.254:41577/ using document root <none> and password file <none>
I20250813 07:53:41.769145 3074 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:41.769409 3074 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:41.769912 3074 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:41.775565 3074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:53:41.776685 3074 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:53:41.784427 3074 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.003s
I20250813 07:53:41.790261 3090 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:41.791422 3074 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250813 07:53:41.791811 3074 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:53:41.792146 3074 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:41.862413 3074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:41.863991 3074 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:41.864498 3074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:41.931818 3074 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:37727
I20250813 07:53:41.931874 3141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:37727 every 8 connection(s)
I20250813 07:53:41.934542 3074 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:53:41.939600 3142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:41.943337 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3074
I20250813 07:53:41.943697 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:53:41.962628 3142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap starting.
I20250813 07:53:41.968355 3142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:41.969995 3142 log.cc:826] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:41.974162 3142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: No bootstrap required, opened a new log
I20250813 07:53:41.990687 3142 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:41.991375 3142 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:41.991601 3142 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Initialized, Role: FOLLOWER
I20250813 07:53:41.992247 3142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:41.992743 3142 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:41.993000 3142 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:41.993288 3142 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:41.997212 3142 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:41.997920 3142 leader_election.cc:304] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0fcd07d0c04d7cbb115a48fcd39a28; no voters:
I20250813 07:53:41.999573 3142 leader_election.cc:290] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:42.000476 3147 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:42.002630 3147 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 1 LEADER]: Becoming Leader. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Running, Role: LEADER
I20250813 07:53:42.003413 3142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:53:42.003360 3147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:42.010596 3148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:53:42.011324 3148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:42.010588 3149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0fcd07d0c04d7cbb115a48fcd39a28. Latest consensus state: current_term: 1 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:53:42.014197 3149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:42.022850 3156 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:53:42.033797 3156 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:53:42.048738 3156 catalog_manager.cc:1349] Generated new cluster ID: e3523349eac14fe09984bb384a73c973
I20250813 07:53:42.049062 3156 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:53:42.070145 3156 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:53:42.072062 3156 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:53:42.095014 3156 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Generated new TSK 0
I20250813 07:53:42.096017 3156 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:53:42.117435 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 07:53:42.436937 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:42.437489 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:42.438009 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:42.469051 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:42.469918 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:53:42.504925 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:42.506259 3166 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:42.507777 3166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:42.519898 3172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:42.529867 3166 server_base.cc:1047] running on GCE node
W20250813 07:53:42.530252 3175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:42.527153 3173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:43.627718 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:43.630290 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:43.631701 3166 hybrid_clock.cc:648] HybridClock initialized: now 1755071623631655 us; error 59 us; skew 500 ppm
I20250813 07:53:43.632485 3166 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:43.638777 3166 webserver.cc:480] Webserver started at http://127.0.106.193:33065/ using document root <none> and password file <none>
I20250813 07:53:43.639662 3166 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:43.639873 3166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:43.640314 3166 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:43.644728 3166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b7eeb2a92c2447f19dc82b6ddcf47232"
format_stamp: "Formatted at 2025-08-13 07:53:43 on dist-test-slave-dr1z"
I20250813 07:53:43.645862 3166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b7eeb2a92c2447f19dc82b6ddcf47232"
format_stamp: "Formatted at 2025-08-13 07:53:43 on dist-test-slave-dr1z"
I20250813 07:53:43.653075 3166 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250813 07:53:43.658963 3182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:43.659994 3166 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250813 07:53:43.660300 3166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b7eeb2a92c2447f19dc82b6ddcf47232"
format_stamp: "Formatted at 2025-08-13 07:53:43 on dist-test-slave-dr1z"
I20250813 07:53:43.660604 3166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:43.728798 3166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:43.730381 3166 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:43.730854 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:43.733831 3166 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:43.738113 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:43.738323 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:43.738576 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:43.738723 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:43.871816 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:41401
I20250813 07:53:43.871920 3294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:41401 every 8 connection(s)
I20250813 07:53:43.874290 3166 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:53:43.876916 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3166
I20250813 07:53:43.877404 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:53:43.883864 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:43.899323 3295 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:53:43.899744 3295 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:43.900751 3295 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:53:43.903587 3107 ts_manager.cc:194] Registered new tserver with Master: b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193:41401)
I20250813 07:53:43.905570 3107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:51515
W20250813 07:53:44.187335 3299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:44.187824 3299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:44.188326 3299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:44.218897 3299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:44.219727 3299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:53:44.254639 3299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:44.255883 3299 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:44.257428 3299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:44.269404 3305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:44.908706 3295 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
W20250813 07:53:44.270174 3306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:44.273324 3308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:45.342090 3307 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250813 07:53:45.342144 3299 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:45.346244 3299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:45.348322 3299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:45.349663 3299 hybrid_clock.cc:648] HybridClock initialized: now 1755071625349624 us; error 49 us; skew 500 ppm
I20250813 07:53:45.350436 3299 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:45.356202 3299 webserver.cc:480] Webserver started at http://127.0.106.194:41813/ using document root <none> and password file <none>
I20250813 07:53:45.357067 3299 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:45.357285 3299 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:45.357756 3299 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:45.362033 3299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1f4b42a1a37b4f2b845b0685441c0091"
format_stamp: "Formatted at 2025-08-13 07:53:45 on dist-test-slave-dr1z"
I20250813 07:53:45.363072 3299 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1f4b42a1a37b4f2b845b0685441c0091"
format_stamp: "Formatted at 2025-08-13 07:53:45 on dist-test-slave-dr1z"
I20250813 07:53:45.369861 3299 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250813 07:53:45.375231 3315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:45.376209 3299 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250813 07:53:45.376524 3299 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1f4b42a1a37b4f2b845b0685441c0091"
format_stamp: "Formatted at 2025-08-13 07:53:45 on dist-test-slave-dr1z"
I20250813 07:53:45.376848 3299 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:45.452448 3299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:45.453924 3299 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:45.454380 3299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:45.456777 3299 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:45.460744 3299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:45.460933 3299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:45.461189 3299 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:45.461335 3299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:45.585717 3299 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:41055
I20250813 07:53:45.585831 3427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:41055 every 8 connection(s)
I20250813 07:53:45.588165 3299 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:53:45.594077 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3299
I20250813 07:53:45.594599 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:53:45.601006 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:45.610008 3428 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:53:45.610532 3428 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:45.611825 3428 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:53:45.614188 3107 ts_manager.cc:194] Registered new tserver with Master: 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:53:45.615433 3107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:51261
W20250813 07:53:45.899456 3432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:45.899981 3432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:45.900483 3432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:45.933614 3432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:45.934434 3432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:53:45.969621 3432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:45.970937 3432 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:45.972527 3432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:45.984210 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:46.619298 3428 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
W20250813 07:53:47.387701 3437 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3432
W20250813 07:53:47.498234 3432 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.514s user 0.616s sys 0.896s
W20250813 07:53:45.985275 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:47.498617 3432 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.514s user 0.616s sys 0.897s
W20250813 07:53:47.500660 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:47.503358 3440 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1515 milliseconds
I20250813 07:53:47.503383 3432 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:47.504508 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:47.506601 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:47.507956 3432 hybrid_clock.cc:648] HybridClock initialized: now 1755071627507927 us; error 35 us; skew 500 ppm
I20250813 07:53:47.508724 3432 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:47.514499 3432 webserver.cc:480] Webserver started at http://127.0.106.195:45785/ using document root <none> and password file <none>
I20250813 07:53:47.515393 3432 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:47.515605 3432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:47.516021 3432 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:53:47.520339 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8d97630ce3164951a12fe25d95ec0dd0"
format_stamp: "Formatted at 2025-08-13 07:53:47 on dist-test-slave-dr1z"
I20250813 07:53:47.521433 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8d97630ce3164951a12fe25d95ec0dd0"
format_stamp: "Formatted at 2025-08-13 07:53:47 on dist-test-slave-dr1z"
I20250813 07:53:47.528249 3432 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 07:53:47.533816 3449 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:47.534888 3432 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250813 07:53:47.535192 3432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8d97630ce3164951a12fe25d95ec0dd0"
format_stamp: "Formatted at 2025-08-13 07:53:47 on dist-test-slave-dr1z"
I20250813 07:53:47.535518 3432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:47.588241 3432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:47.590196 3432 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:47.590749 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:47.593945 3432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:47.597988 3432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:53:47.598181 3432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:47.598434 3432 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:53:47.598598 3432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:47.722288 3432 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:46557
I20250813 07:53:47.722451 3561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:46557 every 8 connection(s)
I20250813 07:53:47.724733 3432 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:53:47.727378 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3432
I20250813 07:53:47.728070 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:53:47.745851 3562 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:53:47.746268 3562 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:47.747434 3562 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:53:47.749428 3107 ts_manager.cc:194] Registered new tserver with Master: 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:53:47.750685 3107 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:55965
I20250813 07:53:47.762928 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:47.797658 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:53:47.798003 427 test_util.cc:276] Using random seed: 1961764320
I20250813 07:53:47.836613 3107 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47750:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 07:53:47.879415 3497 tablet_service.cc:1468] Processing CreateTablet for tablet c6435994656e46f7a213a7f754bfc907 (DEFAULT_TABLE table=TestTable [id=a57516b02b5741f783d5913893bc167b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:47.881356 3497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6435994656e46f7a213a7f754bfc907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:47.900910 3582 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap starting.
I20250813 07:53:47.906580 3582 tablet_bootstrap.cc:654] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:47.908249 3582 log.cc:826] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:47.912796 3582 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: No bootstrap required, opened a new log
I20250813 07:53:47.913190 3582 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250813 07:53:47.930094 3582 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:53:47.930636 3582 raft_consensus.cc:383] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:47.930866 3582 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Initialized, Role: FOLLOWER
I20250813 07:53:47.931522 3582 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:53:47.932006 3582 raft_consensus.cc:397] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:47.932305 3582 raft_consensus.cc:491] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:47.932623 3582 raft_consensus.cc:3058] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:47.937080 3582 raft_consensus.cc:513] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:53:47.938089 3582 leader_election.cc:304] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d97630ce3164951a12fe25d95ec0dd0; no voters:
I20250813 07:53:47.940421 3582 leader_election.cc:290] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:47.941365 3584 raft_consensus.cc:2802] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:47.943111 3562 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:53:47.944675 3584 raft_consensus.cc:695] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 LEADER]: Becoming Leader. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Running, Role: LEADER
I20250813 07:53:47.944926 3582 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent starting tablet: real 0.032s user 0.025s sys 0.007s
I20250813 07:53:47.945540 3584 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:53:47.956991 3107 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195). New cstate: current_term: 1 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:48.168121 427 test_util.cc:276] Using random seed: 1962134428
I20250813 07:53:48.189060 3106 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47752:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 07:53:48.217016 3363 tablet_service.cc:1468] Processing CreateTablet for tablet bf4d758045124a889faf25713edb5904 (DEFAULT_TABLE table=TestTable1 [id=87ec02ff979348959c05518655261ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:48.218415 3363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4d758045124a889faf25713edb5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:48.237425 3603 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap starting.
I20250813 07:53:48.243207 3603 tablet_bootstrap.cc:654] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:48.244952 3603 log.cc:826] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:48.249395 3603 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: No bootstrap required, opened a new log
I20250813 07:53:48.249926 3603 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250813 07:53:48.266723 3603 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:48.267334 3603 raft_consensus.cc:383] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:48.267562 3603 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Initialized, Role: FOLLOWER
I20250813 07:53:48.268225 3603 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:48.268740 3603 raft_consensus.cc:397] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:48.268993 3603 raft_consensus.cc:491] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:48.269290 3603 raft_consensus.cc:3058] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:48.273360 3603 raft_consensus.cc:513] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:48.274122 3603 leader_election.cc:304] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091; no voters:
I20250813 07:53:48.276038 3603 leader_election.cc:290] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:48.276372 3605 raft_consensus.cc:2802] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:48.278650 3605 raft_consensus.cc:695] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 LEADER]: Becoming Leader. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Running, Role: LEADER
I20250813 07:53:48.279584 3605 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:48.280156 3603 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent starting tablet: real 0.030s user 0.022s sys 0.007s
I20250813 07:53:48.290369 3106 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194). New cstate: current_term: 1 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:48.479439 427 test_util.cc:276] Using random seed: 1962445744
I20250813 07:53:48.500188 3102 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47758:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 07:53:48.527874 3230 tablet_service.cc:1468] Processing CreateTablet for tablet 2e81f468bea34fe0a5baee429c75ddb4 (DEFAULT_TABLE table=TestTable2 [id=04ad63c02c004f99999faca40f5d76c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:53:48.529287 3230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e81f468bea34fe0a5baee429c75ddb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:48.548125 3624 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:53:48.553579 3624 tablet_bootstrap.cc:654] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Neither blocks nor log segments found. Creating new log.
I20250813 07:53:48.555259 3624 log.cc:826] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:48.559478 3624 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: No bootstrap required, opened a new log
I20250813 07:53:48.559913 3624 ts_tablet_manager.cc:1397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20250813 07:53:48.577092 3624 raft_consensus.cc:357] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:48.577740 3624 raft_consensus.cc:383] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:53:48.577979 3624 raft_consensus.cc:738] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: FOLLOWER
I20250813 07:53:48.578651 3624 consensus_queue.cc:260] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:48.579156 3624 raft_consensus.cc:397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:48.579407 3624 raft_consensus.cc:491] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:48.579711 3624 raft_consensus.cc:3058] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:53:48.583879 3624 raft_consensus.cc:513] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:48.584633 3624 leader_election.cc:304] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7eeb2a92c2447f19dc82b6ddcf47232; no voters:
I20250813 07:53:48.586639 3624 leader_election.cc:290] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:53:48.587071 3626 raft_consensus.cc:2802] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:53:48.591351 3624 ts_tablet_manager.cc:1428] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.031s user 0.021s sys 0.011s
I20250813 07:53:48.591562 3626 raft_consensus.cc:695] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 LEADER]: Becoming Leader. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Running, Role: LEADER
I20250813 07:53:48.592533 3626 consensus_queue.cc:237] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:48.604460 3102 catalog_manager.cc:5582] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:48.770377 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3074
W20250813 07:53:48.972420 3562 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
W20250813 07:53:49.305503 3428 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
W20250813 07:53:49.621971 3295 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
I20250813 07:53:53.478717 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3166
I20250813 07:53:53.497867 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3299
I20250813 07:53:53.518046 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3432
I20250813 07:53:53.539528 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--webserver_interface=127.0.106.254
--webserver_port=41577
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:37727 with env {}
W20250813 07:53:53.831179 3702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:53.831786 3702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:53.832264 3702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:53.866487 3702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:53:53.866951 3702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:53.867308 3702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:53:53.867637 3702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:53:53.905418 3702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:37727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=41577
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:53.906714 3702 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:53.908259 3702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:53.918759 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:53.919386 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:55.214764 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:55.218153 3710 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1297 milliseconds
W20250813 07:53:55.218417 3702 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.299s user 0.483s sys 0.811s
W20250813 07:53:55.218780 3702 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.300s user 0.483s sys 0.811s
I20250813 07:53:55.219010 3702 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:55.220074 3702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:55.222678 3702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:55.224037 3702 hybrid_clock.cc:648] HybridClock initialized: now 1755071635224007 us; error 48 us; skew 500 ppm
I20250813 07:53:55.224851 3702 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:55.232012 3702 webserver.cc:480] Webserver started at http://127.0.106.254:41577/ using document root <none> and password file <none>
I20250813 07:53:55.232990 3702 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:55.233218 3702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:55.241669 3702 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250813 07:53:55.246905 3718 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:55.248163 3702 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250813 07:53:55.248601 3702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:53:55.250694 3702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:55.314677 3702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:55.316121 3702 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:55.316569 3702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:55.387208 3702 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:37727
I20250813 07:53:55.387271 3769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:37727 every 8 connection(s)
I20250813 07:53:55.390127 3702 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:53:55.393278 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3702
I20250813 07:53:55.395159 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:41401
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=33065
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:55.403612 3770 sys_catalog.cc:263] Verifying existing consensus state
I20250813 07:53:55.410501 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap starting.
I20250813 07:53:55.424187 3770 log.cc:826] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:55.479836 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:53:55.480623 3770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap complete.
I20250813 07:53:55.500849 3770 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:55.502921 3770 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Initialized, Role: FOLLOWER
I20250813 07:53:55.503685 3770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:55.504168 3770 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:55.504431 3770 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:55.504730 3770 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:53:55.510291 3770 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:55.510957 3770 leader_election.cc:304] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0fcd07d0c04d7cbb115a48fcd39a28; no voters:
I20250813 07:53:55.512897 3770 leader_election.cc:290] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250813 07:53:55.513267 3774 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 07:53:55.516260 3774 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 LEADER]: Becoming Leader. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Running, Role: LEADER
I20250813 07:53:55.517159 3774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:53:55.517565 3770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:53:55.524241 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0fcd07d0c04d7cbb115a48fcd39a28. Latest consensus state: current_term: 3 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:53:55.525295 3775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:53:55.526044 3775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:55.527784 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:53:55.539746 3781 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:53:55.551249 3781 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=04ad63c02c004f99999faca40f5d76c2]
I20250813 07:53:55.552883 3781 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=5c7872d244c14f26a61c3557dbe4bf5b]
I20250813 07:53:55.554526 3781 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d8a995806dc843a0b924c5ddaa17ab80]
I20250813 07:53:55.561906 3781 tablet_loader.cc:96] loaded metadata for tablet 2e81f468bea34fe0a5baee429c75ddb4 (table TestTable2 [id=04ad63c02c004f99999faca40f5d76c2])
I20250813 07:53:55.563189 3781 tablet_loader.cc:96] loaded metadata for tablet bf4d758045124a889faf25713edb5904 (table TestTable1 [id=5c7872d244c14f26a61c3557dbe4bf5b])
I20250813 07:53:55.564358 3781 tablet_loader.cc:96] loaded metadata for tablet c6435994656e46f7a213a7f754bfc907 (table TestTable [id=d8a995806dc843a0b924c5ddaa17ab80])
I20250813 07:53:55.566643 3781 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:53:55.571622 3781 catalog_manager.cc:1261] Loaded cluster ID: e3523349eac14fe09984bb384a73c973
I20250813 07:53:55.571938 3781 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:53:55.580714 3781 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:53:55.585983 3781 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Loaded TSK: 0
I20250813 07:53:55.587780 3781 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 07:53:55.743384 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:55.743886 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:55.744372 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:55.776698 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:55.777659 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:53:55.814695 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:41401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=33065
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:55.816064 3772 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:55.817750 3772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:55.830684 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:57.233829 3796 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 3772
W20250813 07:53:57.320156 3796 kernel_stack_watchdog.cc:198] Thread 3772 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 07:53:55.831591 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:57.321571 3772 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.489s user 0.000s sys 0.001s
W20250813 07:53:57.321929 3772 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.489s user 0.001s sys 0.001s
W20250813 07:53:57.327016 3799 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1494 milliseconds
I20250813 07:53:57.327723 3772 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 07:53:57.328178 3801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:57.329342 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:57.331740 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:57.333138 3772 hybrid_clock.cc:648] HybridClock initialized: now 1755071637333077 us; error 71 us; skew 500 ppm
I20250813 07:53:57.333998 3772 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:57.340310 3772 webserver.cc:480] Webserver started at http://127.0.106.193:33065/ using document root <none> and password file <none>
I20250813 07:53:57.341189 3772 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:57.341392 3772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:57.349251 3772 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250813 07:53:57.353956 3808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:57.354990 3772 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:53:57.355290 3772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b7eeb2a92c2447f19dc82b6ddcf47232"
format_stamp: "Formatted at 2025-08-13 07:53:43 on dist-test-slave-dr1z"
I20250813 07:53:57.357141 3772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:57.412792 3772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:57.414251 3772 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:57.414680 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:57.417490 3772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:57.423776 3815 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 07:53:57.434721 3772 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:53:57.434974 3772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250813 07:53:57.435261 3772 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:53:57.439774 3772 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:53:57.439968 3772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250813 07:53:57.440426 3815 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:53:57.508805 3815 log.cc:826] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:57.620852 3815 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:53:57.621857 3815 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:53:57.623042 3815 ts_tablet_manager.cc:1397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.183s user 0.138s sys 0.040s
I20250813 07:53:57.623832 3772 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:41401
I20250813 07:53:57.624099 3922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:41401 every 8 connection(s)
I20250813 07:53:57.627334 3772 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:53:57.632691 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3772
I20250813 07:53:57.634763 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:41055
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=41813
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:57.642688 3815 raft_consensus.cc:357] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:57.645629 3815 raft_consensus.cc:738] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: FOLLOWER
I20250813 07:53:57.646471 3815 consensus_queue.cc:260] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:57.647137 3815 raft_consensus.cc:397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:57.647490 3815 raft_consensus.cc:491] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:57.649317 3815 raft_consensus.cc:3058] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:53:57.657035 3815 raft_consensus.cc:513] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:57.658118 3815 leader_election.cc:304] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7eeb2a92c2447f19dc82b6ddcf47232; no voters:
I20250813 07:53:57.665565 3815 leader_election.cc:290] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 07:53:57.666394 3928 raft_consensus.cc:2802] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:53:57.674436 3923 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:53:57.674934 3923 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:57.676383 3923 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:53:57.676677 3815 ts_tablet_manager.cc:1428] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.053s user 0.037s sys 0.009s
I20250813 07:53:57.677764 3928 raft_consensus.cc:695] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEADER]: Becoming Leader. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Running, Role: LEADER
I20250813 07:53:57.678599 3928 consensus_queue.cc:237] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:53:57.690958 3735 ts_manager.cc:194] Registered new tserver with Master: b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193:41401)
I20250813 07:53:57.695181 3735 catalog_manager.cc:5582] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:57.742358 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:52557
I20250813 07:53:57.746742 3923 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
W20250813 07:53:57.957618 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:57.958129 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:57.958614 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:57.990281 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:57.991194 3927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:53:58.026492 3927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:41055
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=41813
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:58.027791 3927 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:58.029364 3927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:58.041016 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:58.041900 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:53:59.141413 3944 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250813 07:53:59.142705 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:59.142753 3927 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:53:59.145826 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:53:59.148279 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:53:59.149672 3927 hybrid_clock.cc:648] HybridClock initialized: now 1755071639149635 us; error 49 us; skew 500 ppm
I20250813 07:53:59.150446 3927 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:53:59.156445 3927 webserver.cc:480] Webserver started at http://127.0.106.194:41813/ using document root <none> and password file <none>
I20250813 07:53:59.157298 3927 fs_manager.cc:362] Metadata directory not provided
I20250813 07:53:59.157538 3927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:53:59.164927 3927 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250813 07:53:59.169381 3952 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:53:59.170410 3927 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250813 07:53:59.170717 3927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1f4b42a1a37b4f2b845b0685441c0091"
format_stamp: "Formatted at 2025-08-13 07:53:45 on dist-test-slave-dr1z"
I20250813 07:53:59.172583 3927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:53:59.219832 3927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:53:59.221237 3927 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:53:59.221673 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:53:59.224112 3927 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:53:59.229635 3959 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 07:53:59.236930 3927 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:53:59.237195 3927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250813 07:53:59.237511 3927 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:53:59.242162 3927 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:53:59.242357 3927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.003s
I20250813 07:53:59.242937 3959 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap starting.
I20250813 07:53:59.295991 3959 log.cc:826] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Log is configured to *not* fsync() on all Append() calls
I20250813 07:53:59.367033 3959 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:53:59.367820 3959 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap complete.
I20250813 07:53:59.369316 3959 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent bootstrapping tablet: real 0.127s user 0.085s sys 0.036s
I20250813 07:53:59.383616 3959 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:59.386618 3959 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Initialized, Role: FOLLOWER
I20250813 07:53:59.387646 3959 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:59.388317 3959 raft_consensus.cc:397] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:53:59.388657 3959 raft_consensus.cc:491] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:53:59.389055 3959 raft_consensus.cc:3058] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:53:59.397723 3959 raft_consensus.cc:513] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:59.398732 3959 leader_election.cc:304] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091; no voters:
I20250813 07:53:59.406936 3959 leader_election.cc:290] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 07:53:59.408727 4053 raft_consensus.cc:2802] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:53:59.411263 4053 raft_consensus.cc:695] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEADER]: Becoming Leader. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Running, Role: LEADER
I20250813 07:53:59.412609 4053 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } }
I20250813 07:53:59.418802 3959 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent starting tablet: real 0.049s user 0.025s sys 0.016s
I20250813 07:53:59.424561 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:41055
I20250813 07:53:59.425024 4071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:41055 every 8 connection(s)
I20250813 07:53:59.426993 3927 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:53:59.432088 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 3927
I20250813 07:53:59.433939 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:46557
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=45785
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:53:59.444108 4072 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:53:59.444530 4072 heartbeater.cc:461] Registering TS with master...
I20250813 07:53:59.445518 4072 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:53:59.448892 3735 ts_manager.cc:194] Registered new tserver with Master: 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:53:59.450177 3735 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: term changed from 0 to 2, leader changed from <none> to 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194), VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) added. New cstate: current_term: 2 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } }
I20250813 07:53:59.464355 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:49471
I20250813 07:53:59.469017 4072 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:53:59.483121 4022 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:53:59.486308 4053 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } } }
I20250813 07:53:59.493513 3721 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf4d758045124a889faf25713edb5904 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250813 07:53:59.496364 3734 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: config changed from index -1 to 8, NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) added. New cstate: current_term: 2 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250813 07:53:59.500643 3956 consensus_peers.cc:489] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 -> Peer b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193:41401): Couldn't send request to peer b7eeb2a92c2447f19dc82b6ddcf47232. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf4d758045124a889faf25713edb5904. This is attempt 1: this message will repeat every 5th retry.
W20250813 07:53:59.504937 3734 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf4d758045124a889faf25713edb5904 with cas_config_opid_index 8: no extra replica candidate found for tablet bf4d758045124a889faf25713edb5904 (table TestTable1 [id=5c7872d244c14f26a61c3557dbe4bf5b]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250813 07:53:59.746662 4076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:53:59.747125 4076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:53:59.747560 4076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:53:59.779502 4076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:53:59.780339 4076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:53:59.814666 4076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:46557
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=45785
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:53:59.815868 4076 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:53:59.817396 4076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:53:59.828704 4089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:53:59.880779 4095 ts_tablet_manager.cc:927] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Initiating tablet copy from peer 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:53:59.882822 4095 tablet_copy_client.cc:323] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.194:41055
I20250813 07:53:59.893224 4042 tablet_copy_service.cc:140] P 1f4b42a1a37b4f2b845b0685441c0091: Received BeginTabletCopySession request for tablet bf4d758045124a889faf25713edb5904 from peer b7eeb2a92c2447f19dc82b6ddcf47232 ({username='slave'} at 127.0.106.193:49307)
I20250813 07:53:59.893697 4042 tablet_copy_service.cc:161] P 1f4b42a1a37b4f2b845b0685441c0091: Beginning new tablet copy session on tablet bf4d758045124a889faf25713edb5904 from peer b7eeb2a92c2447f19dc82b6ddcf47232 at {username='slave'} at 127.0.106.193:49307: session id = b7eeb2a92c2447f19dc82b6ddcf47232-bf4d758045124a889faf25713edb5904
I20250813 07:53:59.898590 4042 tablet_copy_source_session.cc:215] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 07:53:59.902309 4095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4d758045124a889faf25713edb5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:53:59.912283 4095 tablet_copy_client.cc:806] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Starting download of 0 data blocks...
I20250813 07:53:59.912789 4095 tablet_copy_client.cc:670] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Starting download of 1 WAL segments...
I20250813 07:53:59.917054 4095 tablet_copy_client.cc:538] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 07:53:59.922454 4095 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:53:59.993587 4095 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:53:59.994190 4095 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:53:59.994616 4095 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.072s user 0.063s sys 0.009s
I20250813 07:53:59.996258 4095 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:53:59.996663 4095 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: LEARNER
I20250813 07:53:59.997141 4095 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:00.000998 4095 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250813 07:54:00.002630 4042 tablet_copy_service.cc:342] P 1f4b42a1a37b4f2b845b0685441c0091: Request end of tablet copy session b7eeb2a92c2447f19dc82b6ddcf47232-bf4d758045124a889faf25713edb5904 received from {username='slave'} at 127.0.106.193:49307
I20250813 07:54:00.002971 4042 tablet_copy_service.cc:434] P 1f4b42a1a37b4f2b845b0685441c0091: ending tablet copy session b7eeb2a92c2447f19dc82b6ddcf47232-bf4d758045124a889faf25713edb5904 on tablet bf4d758045124a889faf25713edb5904 with peer b7eeb2a92c2447f19dc82b6ddcf47232
I20250813 07:54:00.455929 3878 raft_consensus.cc:1215] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20250813 07:54:00.849076 4102 raft_consensus.cc:1062] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: attempting to promote NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 to VOTER
I20250813 07:54:00.850518 4102 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:00.855387 3878 raft_consensus.cc:1273] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Refusing update from remote peer 1f4b42a1a37b4f2b845b0685441c0091: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250813 07:54:00.856467 4101 consensus_queue.cc:1035] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250813 07:54:00.862923 4102 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:00.864439 3878 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:00.870846 3735 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: config changed from index 8 to 9, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250813 07:53:59.829981 4090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:00.926110 4091 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250813 07:54:00.927561 4092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:00.927553 4076 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:00.930378 4076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:00.932761 4076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:00.934175 4076 hybrid_clock.cc:648] HybridClock initialized: now 1755071640934127 us; error 49 us; skew 500 ppm
I20250813 07:54:00.934993 4076 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:00.941323 4076 webserver.cc:480] Webserver started at http://127.0.106.195:45785/ using document root <none> and password file <none>
I20250813 07:54:00.942267 4076 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:00.942484 4076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:00.950294 4076 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250813 07:54:00.954813 4113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:00.955885 4076 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250813 07:54:00.956220 4076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8d97630ce3164951a12fe25d95ec0dd0"
format_stamp: "Formatted at 2025-08-13 07:53:47 on dist-test-slave-dr1z"
I20250813 07:54:00.958199 4076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:01.023185 4076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:01.024596 4076 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:01.025008 4076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:01.027585 4076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:01.033218 4120 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 07:54:01.040854 4076 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:54:01.041080 4076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250813 07:54:01.041347 4076 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:54:01.046068 4076 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:54:01.046258 4076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250813 07:54:01.046679 4120 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap starting.
I20250813 07:54:01.104887 4120 log.cc:826] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:01.198314 4120 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:01.199087 4120 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap complete.
I20250813 07:54:01.200443 4120 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent bootstrapping tablet: real 0.154s user 0.079s sys 0.071s
I20250813 07:54:01.211338 4076 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:46557
I20250813 07:54:01.211514 4227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:46557 every 8 connection(s)
I20250813 07:54:01.214025 4076 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:54:01.216835 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4076
I20250813 07:54:01.215744 4120 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:54:01.218798 4120 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Initialized, Role: FOLLOWER
I20250813 07:54:01.219682 4120 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:54:01.220467 4120 raft_consensus.cc:397] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:01.220811 4120 raft_consensus.cc:491] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:01.221256 4120 raft_consensus.cc:3058] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:54:01.229450 4120 raft_consensus.cc:513] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:54:01.230414 4120 leader_election.cc:304] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d97630ce3164951a12fe25d95ec0dd0; no voters:
I20250813 07:54:01.233923 4120 leader_election.cc:290] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 07:54:01.234591 4234 raft_consensus.cc:2802] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:54:01.246172 4228 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:54:01.246717 4228 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:01.248023 4228 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:54:01.250496 4234 raft_consensus.cc:695] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: Becoming Leader. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Running, Role: LEADER
I20250813 07:54:01.251057 4120 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent starting tablet: real 0.050s user 0.041s sys 0.009s
I20250813 07:54:01.251392 4234 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } }
I20250813 07:54:01.261736 3735 ts_manager.cc:194] Registered new tserver with Master: 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:54:01.262746 3735 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 reported cstate change: term changed from 0 to 2, leader changed from <none> to 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195), VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) added. New cstate: current_term: 2 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: HEALTHY } } }
I20250813 07:54:01.264397 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:54:01.268801 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:54:01.272058 3735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:35185
W20250813 07:54:01.272032 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250813 07:54:01.275553 4228 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:54:01.286114 4183 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } }
I20250813 07:54:01.289338 4236 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } }
I20250813 07:54:01.296622 3721 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6435994656e46f7a213a7f754bfc907 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250813 07:54:01.299348 4116 consensus_peers.cc:489] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 -> Peer 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055): Couldn't send request to peer 1f4b42a1a37b4f2b845b0685441c0091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6435994656e46f7a213a7f754bfc907. This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:01.299307 3735 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 reported cstate change: config changed from index -1 to 10, NON_VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) added. New cstate: current_term: 2 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250813 07:54:01.308302 4183 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:01.311353 4236 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } } }
W20250813 07:54:01.313004 4116 consensus_peers.cc:489] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 -> Peer 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055): Couldn't send request to peer 1f4b42a1a37b4f2b845b0685441c0091. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6435994656e46f7a213a7f754bfc907. This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:01.318516 3721 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6435994656e46f7a213a7f754bfc907 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250813 07:54:01.321369 3735 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 reported cstate change: config changed from index 10 to 11, NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) added. New cstate: current_term: 2 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250813 07:54:01.322113 4117 consensus_peers.cc:489] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 -> Peer b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193:41401): Couldn't send request to peer b7eeb2a92c2447f19dc82b6ddcf47232. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6435994656e46f7a213a7f754bfc907. This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:01.498726 4022 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } }
I20250813 07:54:01.503965 3878 raft_consensus.cc:1273] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Refusing update from remote peer 1f4b42a1a37b4f2b845b0685441c0091: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250813 07:54:01.505070 4243 consensus_queue.cc:1035] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250813 07:54:01.509688 4244 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } } }
W20250813 07:54:01.511492 3955 consensus_peers.cc:489] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 -> Peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Couldn't send request to peer 8d97630ce3164951a12fe25d95ec0dd0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf4d758045124a889faf25713edb5904. This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:01.511036 3878 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } } }
I20250813 07:54:01.516055 3721 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf4d758045124a889faf25713edb5904 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250813 07:54:01.519074 3735 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: config changed from index 9 to 10, NON_VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) added. New cstate: current_term: 2 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250813 07:54:01.803560 3720 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf4d758045124a889faf25713edb5904 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250813 07:54:01.807538 4247 ts_tablet_manager.cc:927] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Initiating tablet copy from peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:54:01.809499 4247 tablet_copy_client.cc:323] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.195:46557
I20250813 07:54:01.811167 4203 tablet_copy_service.cc:140] P 8d97630ce3164951a12fe25d95ec0dd0: Received BeginTabletCopySession request for tablet c6435994656e46f7a213a7f754bfc907 from peer 1f4b42a1a37b4f2b845b0685441c0091 ({username='slave'} at 127.0.106.194:42013)
I20250813 07:54:01.811609 4203 tablet_copy_service.cc:161] P 8d97630ce3164951a12fe25d95ec0dd0: Beginning new tablet copy session on tablet c6435994656e46f7a213a7f754bfc907 from peer 1f4b42a1a37b4f2b845b0685441c0091 at {username='slave'} at 127.0.106.194:42013: session id = 1f4b42a1a37b4f2b845b0685441c0091-c6435994656e46f7a213a7f754bfc907
I20250813 07:54:01.816035 4203 tablet_copy_source_session.cc:215] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 07:54:01.819278 4247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6435994656e46f7a213a7f754bfc907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:01.829483 4247 tablet_copy_client.cc:806] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: tablet copy: Starting download of 0 data blocks...
I20250813 07:54:01.829965 4247 tablet_copy_client.cc:670] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: tablet copy: Starting download of 1 WAL segments...
I20250813 07:54:01.833506 4247 tablet_copy_client.cc:538] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 07:54:01.838644 4247 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap starting.
I20250813 07:54:01.887105 4250 ts_tablet_manager.cc:927] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Initiating tablet copy from peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:54:01.888486 4250 tablet_copy_client.cc:323] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.195:46557
I20250813 07:54:01.898135 4203 tablet_copy_service.cc:140] P 8d97630ce3164951a12fe25d95ec0dd0: Received BeginTabletCopySession request for tablet c6435994656e46f7a213a7f754bfc907 from peer b7eeb2a92c2447f19dc82b6ddcf47232 ({username='slave'} at 127.0.106.193:38493)
I20250813 07:54:01.898531 4203 tablet_copy_service.cc:161] P 8d97630ce3164951a12fe25d95ec0dd0: Beginning new tablet copy session on tablet c6435994656e46f7a213a7f754bfc907 from peer b7eeb2a92c2447f19dc82b6ddcf47232 at {username='slave'} at 127.0.106.193:38493: session id = b7eeb2a92c2447f19dc82b6ddcf47232-c6435994656e46f7a213a7f754bfc907
I20250813 07:54:01.902558 4203 tablet_copy_source_session.cc:215] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 07:54:01.905189 4250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6435994656e46f7a213a7f754bfc907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:01.919714 4250 tablet_copy_client.cc:806] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Starting download of 0 data blocks...
I20250813 07:54:01.920297 4250 tablet_copy_client.cc:670] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Starting download of 1 WAL segments...
I20250813 07:54:01.923660 4250 tablet_copy_client.cc:538] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 07:54:01.931634 4250 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:54:01.948182 4247 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:01.949005 4247 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap complete.
I20250813 07:54:01.949613 4247 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent bootstrapping tablet: real 0.111s user 0.090s sys 0.020s
I20250813 07:54:01.951889 4247 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:01.952557 4247 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Initialized, Role: LEARNER
I20250813 07:54:01.953084 4247 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:01.955638 4247 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250813 07:54:01.957178 4203 tablet_copy_service.cc:342] P 8d97630ce3164951a12fe25d95ec0dd0: Request end of tablet copy session 1f4b42a1a37b4f2b845b0685441c0091-c6435994656e46f7a213a7f754bfc907 received from {username='slave'} at 127.0.106.194:42013
I20250813 07:54:01.957710 4203 tablet_copy_service.cc:434] P 8d97630ce3164951a12fe25d95ec0dd0: ending tablet copy session 1f4b42a1a37b4f2b845b0685441c0091-c6435994656e46f7a213a7f754bfc907 on tablet c6435994656e46f7a213a7f754bfc907 with peer 1f4b42a1a37b4f2b845b0685441c0091
I20250813 07:54:01.976382 4254 ts_tablet_manager.cc:927] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Initiating tablet copy from peer 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:54:01.978386 4254 tablet_copy_client.cc:323] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.106.194:41055
I20250813 07:54:01.979930 4042 tablet_copy_service.cc:140] P 1f4b42a1a37b4f2b845b0685441c0091: Received BeginTabletCopySession request for tablet bf4d758045124a889faf25713edb5904 from peer 8d97630ce3164951a12fe25d95ec0dd0 ({username='slave'} at 127.0.106.195:35445)
I20250813 07:54:01.980356 4042 tablet_copy_service.cc:161] P 1f4b42a1a37b4f2b845b0685441c0091: Beginning new tablet copy session on tablet bf4d758045124a889faf25713edb5904 from peer 8d97630ce3164951a12fe25d95ec0dd0 at {username='slave'} at 127.0.106.195:35445: session id = 8d97630ce3164951a12fe25d95ec0dd0-bf4d758045124a889faf25713edb5904
I20250813 07:54:01.984846 4042 tablet_copy_source_session.cc:215] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 07:54:01.987915 4254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4d758045124a889faf25713edb5904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:02.000443 4254 tablet_copy_client.cc:806] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: tablet copy: Starting download of 0 data blocks...
I20250813 07:54:02.000990 4254 tablet_copy_client.cc:670] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: tablet copy: Starting download of 1 WAL segments...
I20250813 07:54:02.005112 4254 tablet_copy_client.cc:538] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 07:54:02.011173 4254 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap starting.
I20250813 07:54:02.035378 4250 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:02.036093 4250 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:54:02.036514 4250 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.105s user 0.088s sys 0.016s
I20250813 07:54:02.038085 4250 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:02.038558 4250 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: LEARNER
I20250813 07:54:02.038954 4250 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: NON_VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: true } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:02.040982 4250 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250813 07:54:02.042649 4203 tablet_copy_service.cc:342] P 8d97630ce3164951a12fe25d95ec0dd0: Request end of tablet copy session b7eeb2a92c2447f19dc82b6ddcf47232-c6435994656e46f7a213a7f754bfc907 received from {username='slave'} at 127.0.106.193:38493
I20250813 07:54:02.043027 4203 tablet_copy_service.cc:434] P 8d97630ce3164951a12fe25d95ec0dd0: ending tablet copy session b7eeb2a92c2447f19dc82b6ddcf47232-c6435994656e46f7a213a7f754bfc907 on tablet c6435994656e46f7a213a7f754bfc907 with peer b7eeb2a92c2447f19dc82b6ddcf47232
I20250813 07:54:02.085130 4254 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:02.085764 4254 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap complete.
I20250813 07:54:02.086194 4254 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent bootstrapping tablet: real 0.075s user 0.067s sys 0.008s
I20250813 07:54:02.088182 4254 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } }
I20250813 07:54:02.088670 4254 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Initialized, Role: LEARNER
I20250813 07:54:02.089082 4254 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: NON_VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: true } }
I20250813 07:54:02.090512 4254 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250813 07:54:02.091917 4042 tablet_copy_service.cc:342] P 1f4b42a1a37b4f2b845b0685441c0091: Request end of tablet copy session 8d97630ce3164951a12fe25d95ec0dd0-bf4d758045124a889faf25713edb5904 received from {username='slave'} at 127.0.106.195:35445
I20250813 07:54:02.092195 4042 tablet_copy_service.cc:434] P 1f4b42a1a37b4f2b845b0685441c0091: ending tablet copy session 8d97630ce3164951a12fe25d95ec0dd0-bf4d758045124a889faf25713edb5904 on tablet bf4d758045124a889faf25713edb5904 with peer 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:02.186983 4022 raft_consensus.cc:1215] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250813 07:54:02.276587 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b7eeb2a92c2447f19dc82b6ddcf47232 to finish bootstrapping
I20250813 07:54:02.289249 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1f4b42a1a37b4f2b845b0685441c0091 to finish bootstrapping
I20250813 07:54:02.300539 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8d97630ce3164951a12fe25d95ec0dd0 to finish bootstrapping
I20250813 07:54:02.324718 3878 raft_consensus.cc:1215] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250813 07:54:02.407995 4183 raft_consensus.cc:1215] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250813 07:54:02.574299 4002 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:02.578747 4163 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:02.580538 3858 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:02.692224 4259 raft_consensus.cc:1062] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: attempting to promote NON_VOTER 1f4b42a1a37b4f2b845b0685441c0091 to VOTER
I20250813 07:54:02.694399 4259 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } }
I20250813 07:54:02.710594 3878 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Refusing update from remote peer 8d97630ce3164951a12fe25d95ec0dd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250813 07:54:02.712066 4234 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250813 07:54:02.718822 4022 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEARNER]: Refusing update from remote peer 8d97630ce3164951a12fe25d95ec0dd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250813 07:54:02.721386 4234 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250813 07:54:02.725001 4292 raft_consensus.cc:1025] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: attempt to promote peer b7eeb2a92c2447f19dc82b6ddcf47232: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250813 07:54:02.740528 4292 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } } }
I20250813 07:54:02.755815 3878 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } } }
I20250813 07:54:02.759636 3734 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 reported cstate change: config changed from index 11 to 12, 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250813 07:54:02.766492 4258 raft_consensus.cc:1062] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: attempting to promote NON_VOTER b7eeb2a92c2447f19dc82b6ddcf47232 to VOTER
I20250813 07:54:02.768296 4258 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:02.776738 3878 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 LEARNER]: Refusing update from remote peer 8d97630ce3164951a12fe25d95ec0dd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250813 07:54:02.778369 4259 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250813 07:54:02.788213 3878 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:02.793933 4292 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:02.750903 4022 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: NON_VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: true } } }
I20250813 07:54:02.819020 4021 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Refusing update from remote peer 8d97630ce3164951a12fe25d95ec0dd0: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250813 07:54:02.821727 3735 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: config changed from index 12 to 13, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8d97630ce3164951a12fe25d95ec0dd0" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:02.829279 4259 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5a0fcd07d0c04d7cbb115a48fcd39a28 | 127.0.106.254:37727 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
I20250813 07:54:02.835601 4021 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:46831 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
1f4b42a1a37b4f2b845b0685441c0091 | 127.0.106.194:41055 | HEALTHY | <none> | 1 | 0
8d97630ce3164951a12fe25d95ec0dd0 | 127.0.106.195:46557 | HEALTHY | <none> | 1 | 0
b7eeb2a92c2447f19dc82b6ddcf47232 | 127.0.106.193:41401 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:41401
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:41055
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:46557
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.106.193:41401
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.106.194:41055
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.106.195:46557
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:46831 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 07:54:02.850435 427 log_verifier.cc:126] Checking tablet 2e81f468bea34fe0a5baee429c75ddb4
I20250813 07:54:02.880602 427 log_verifier.cc:177] Verified matching terms for 8 ops in tablet 2e81f468bea34fe0a5baee429c75ddb4
I20250813 07:54:02.880841 427 log_verifier.cc:126] Checking tablet bf4d758045124a889faf25713edb5904
I20250813 07:54:02.963433 427 log_verifier.cc:177] Verified matching terms for 10 ops in tablet bf4d758045124a889faf25713edb5904
I20250813 07:54:02.963663 427 log_verifier.cc:126] Checking tablet c6435994656e46f7a213a7f754bfc907
I20250813 07:54:03.000013 4271 raft_consensus.cc:1062] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: attempting to promote NON_VOTER 8d97630ce3164951a12fe25d95ec0dd0 to VOTER
I20250813 07:54:03.001951 4271 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:03.008162 4183 raft_consensus.cc:1273] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 LEARNER]: Refusing update from remote peer 1f4b42a1a37b4f2b845b0685441c0091: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250813 07:54:03.008311 3878 raft_consensus.cc:1273] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Refusing update from remote peer 1f4b42a1a37b4f2b845b0685441c0091: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250813 07:54:03.010068 4298 consensus_queue.cc:1035] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250813 07:54:03.011876 4298 consensus_queue.cc:1035] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250813 07:54:03.018422 4298 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } } }
I20250813 07:54:03.020916 3878 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } } }
I20250813 07:54:03.023101 4183 raft_consensus.cc:2953] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } } }
I20250813 07:54:03.031692 3734 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: config changed from index 10 to 11, 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1f4b42a1a37b4f2b845b0685441c0091" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 07:54:03.067348 427 log_verifier.cc:177] Verified matching terms for 13 ops in tablet c6435994656e46f7a213a7f754bfc907
I20250813 07:54:03.067727 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3702
I20250813 07:54:03.094054 427 minidump.cc:252] Setting minidump size limit to 20M
I20250813 07:54:03.095274 427 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:03.096295 427 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:03.107024 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:03.108083 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:03.107282 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:03.108320 427 server_base.cc:1047] running on GCE node
I20250813 07:54:03.206146 427 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250813 07:54:03.206403 427 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250813 07:54:03.206544 427 hybrid_clock.cc:648] HybridClock initialized: now 1755071643206530 us; error 0 us; skew 500 ppm
I20250813 07:54:03.207136 427 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:03.211869 427 webserver.cc:480] Webserver started at http://0.0.0.0:45129/ using document root <none> and password file <none>
I20250813 07:54:03.212634 427 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:03.212821 427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:03.217922 427 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250813 07:54:03.221427 4313 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:03.222415 427 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250813 07:54:03.222716 427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:54:03.224334 427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:03.241776 427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:03.243261 427 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:03.243683 427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:03.252110 427 sys_catalog.cc:263] Verifying existing consensus state
W20250813 07:54:03.255704 427 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.0.106.254:37727 exists but no master address supplied!
I20250813 07:54:03.257628 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap starting.
I20250813 07:54:03.298215 427 log.cc:826] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Log is configured to *not* fsync() on all Append() calls
W20250813 07:54:03.311801 4228 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
I20250813 07:54:03.363879 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:03.364809 427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap complete.
I20250813 07:54:03.377748 427 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:03.378309 427 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Initialized, Role: FOLLOWER
I20250813 07:54:03.379086 427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:03.379604 427 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:03.379828 427 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:03.380110 427 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 3 FOLLOWER]: Advancing to term 4
I20250813 07:54:03.385443 427 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:03.386221 427 leader_election.cc:304] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0fcd07d0c04d7cbb115a48fcd39a28; no voters:
I20250813 07:54:03.387317 427 leader_election.cc:290] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250813 07:54:03.387662 4321 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 4 FOLLOWER]: Leader election won for term 4
I20250813 07:54:03.388940 4321 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 4 LEADER]: Becoming Leader. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Running, Role: LEADER
I20250813 07:54:03.389765 4321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:03.396117 4323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0fcd07d0c04d7cbb115a48fcd39a28. Latest consensus state: current_term: 4 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:54:03.396651 4323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:03.397679 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:54:03.398114 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:03.425004 427 tablet_replica.cc:331] stopping tablet replica
I20250813 07:54:03.425531 427 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 4 LEADER]: Raft consensus shutting down.
I20250813 07:54:03.426072 427 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250813 07:54:03.428401 427 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250813 07:54:03.428891 427 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250813 07:54:03.459270 427 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250813 07:54:04.059731 4072 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
W20250813 07:54:04.071300 3923 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:37727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:37727: connect: Connection refused (error 111)
I20250813 07:54:08.556638 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3772
I20250813 07:54:08.578729 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 3927
I20250813 07:54:08.602852 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4076
I20250813 07:54:08.628388 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--webserver_interface=127.0.106.254
--webserver_port=41577
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:37727 with env {}
W20250813 07:54:08.922196 4394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:08.922763 4394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:08.923213 4394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:08.954643 4394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:54:08.954990 4394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:08.955293 4394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:54:08.955538 4394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:54:09.006387 4394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:37727
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=41577
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:09.007754 4394 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:09.009346 4394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:09.020255 4400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:09.020545 4401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:10.121650 4402 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1099 milliseconds
W20250813 07:54:10.122622 4403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:10.125350 4394 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.105s user 0.369s sys 0.736s
W20250813 07:54:10.125680 4394 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.106s user 0.369s sys 0.736s
I20250813 07:54:10.125931 4394 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:10.127031 4394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:10.129628 4394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:10.130992 4394 hybrid_clock.cc:648] HybridClock initialized: now 1755071650130963 us; error 41 us; skew 500 ppm
I20250813 07:54:10.131793 4394 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:10.139122 4394 webserver.cc:480] Webserver started at http://127.0.106.254:41577/ using document root <none> and password file <none>
I20250813 07:54:10.139995 4394 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:10.140213 4394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:10.148370 4394 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250813 07:54:10.153227 4410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:10.154392 4394 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:54:10.154747 4394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28"
format_stamp: "Formatted at 2025-08-13 07:53:41 on dist-test-slave-dr1z"
I20250813 07:54:10.156659 4394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:10.213662 4394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:10.215085 4394 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:10.215529 4394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:10.283762 4394 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:37727
I20250813 07:54:10.283834 4461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:37727 every 8 connection(s)
I20250813 07:54:10.286535 4394 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:54:10.293843 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4394
I20250813 07:54:10.295699 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:41401
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=33065
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:54:10.297362 4462 sys_catalog.cc:263] Verifying existing consensus state
I20250813 07:54:10.303974 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap starting.
I20250813 07:54:10.314519 4462 log.cc:826] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:10.389972 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:10.390707 4462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Bootstrap complete.
I20250813 07:54:10.409111 4462 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:10.411099 4462 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Initialized, Role: FOLLOWER
I20250813 07:54:10.411763 4462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:10.412220 4462 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:10.412458 4462 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:10.412724 4462 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 5 FOLLOWER]: Advancing to term 6
I20250813 07:54:10.417624 4462 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:10.418210 4462 leader_election.cc:304] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0fcd07d0c04d7cbb115a48fcd39a28; no voters:
I20250813 07:54:10.420066 4462 leader_election.cc:290] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250813 07:54:10.420323 4466 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 6 FOLLOWER]: Leader election won for term 6
I20250813 07:54:10.423962 4466 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [term 6 LEADER]: Becoming Leader. State: Replica: 5a0fcd07d0c04d7cbb115a48fcd39a28, State: Running, Role: LEADER
I20250813 07:54:10.424755 4466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } }
I20250813 07:54:10.425242 4462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:54:10.431588 4468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0fcd07d0c04d7cbb115a48fcd39a28. Latest consensus state: current_term: 6 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:54:10.433353 4468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:10.432793 4467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0fcd07d0c04d7cbb115a48fcd39a28" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37727 } } }
I20250813 07:54:10.433740 4467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:10.438573 4473 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:54:10.451237 4473 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=04ad63c02c004f99999faca40f5d76c2]
I20250813 07:54:10.452916 4473 catalog_manager.cc:671] Loaded metadata for table TestTable [id=2320e9785f604267b57e056dfe72d135]
I20250813 07:54:10.454457 4473 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=5c7872d244c14f26a61c3557dbe4bf5b]
I20250813 07:54:10.462147 4473 tablet_loader.cc:96] loaded metadata for tablet 2e81f468bea34fe0a5baee429c75ddb4 (table TestTable2 [id=04ad63c02c004f99999faca40f5d76c2])
I20250813 07:54:10.464334 4473 tablet_loader.cc:96] loaded metadata for tablet bf4d758045124a889faf25713edb5904 (table TestTable1 [id=5c7872d244c14f26a61c3557dbe4bf5b])
I20250813 07:54:10.465600 4473 tablet_loader.cc:96] loaded metadata for tablet c6435994656e46f7a213a7f754bfc907 (table TestTable [id=2320e9785f604267b57e056dfe72d135])
I20250813 07:54:10.466921 4473 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:54:10.472239 4473 catalog_manager.cc:1261] Loaded cluster ID: e3523349eac14fe09984bb384a73c973
I20250813 07:54:10.472607 4473 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:54:10.481442 4473 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:54:10.486884 4473 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 5a0fcd07d0c04d7cbb115a48fcd39a28: Loaded TSK: 0
I20250813 07:54:10.488490 4473 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 07:54:10.625284 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:10.625799 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:10.626286 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:10.658879 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:10.659723 4464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:54:10.694135 4464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:41401
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=33065
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:10.695410 4464 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:10.696944 4464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:10.709493 4489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:10.710078 4490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:12.045608 4492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:12.046053 4464 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.335s user 0.004s sys 0.004s
W20250813 07:54:12.046386 4464 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.336s user 0.004s sys 0.004s
I20250813 07:54:12.046645 4464 server_base.cc:1047] running on GCE node
I20250813 07:54:12.047930 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:12.050750 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:12.052290 4464 hybrid_clock.cc:648] HybridClock initialized: now 1755071652052221 us; error 74 us; skew 500 ppm
I20250813 07:54:12.053371 4464 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:12.062021 4464 webserver.cc:480] Webserver started at http://127.0.106.193:33065/ using document root <none> and password file <none>
I20250813 07:54:12.063225 4464 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:12.063503 4464 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:12.074925 4464 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.005s
I20250813 07:54:12.081173 4499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:12.082556 4464 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250813 07:54:12.082945 4464 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b7eeb2a92c2447f19dc82b6ddcf47232"
format_stamp: "Formatted at 2025-08-13 07:53:43 on dist-test-slave-dr1z"
I20250813 07:54:12.085721 4464 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:12.174192 4464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:12.175631 4464 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:12.176038 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:12.178646 4464 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:12.184677 4506 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250813 07:54:12.199815 4464 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250813 07:54:12.200038 4464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.002s sys 0.001s
I20250813 07:54:12.200301 4464 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250813 07:54:12.205502 4506 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:54:12.214712 4464 ts_tablet_manager.cc:610] Registered 3 tablets
I20250813 07:54:12.214989 4464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.015s user 0.010s sys 0.003s
I20250813 07:54:12.277443 4506 log.cc:826] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:12.419071 4464 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:41401
I20250813 07:54:12.419343 4506 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:12.419351 4613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:41401 every 8 connection(s)
I20250813 07:54:12.420521 4506 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:54:12.421975 4464 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:54:12.422111 4506 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.217s user 0.186s sys 0.023s
I20250813 07:54:12.431912 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4464
I20250813 07:54:12.433961 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:41055
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=41813
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:54:12.446178 4506 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:12.449656 4506 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: FOLLOWER
I20250813 07:54:12.450707 4506 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:12.458973 4506 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.037s user 0.033s sys 0.001s
I20250813 07:54:12.459759 4506 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:54:12.479691 4614 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:54:12.480374 4614 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:12.482192 4614 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:54:12.489050 4427 ts_manager.cc:194] Registered new tserver with Master: b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193:41401)
I20250813 07:54:12.494325 4427 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) added, VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) added, VOTER b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:12.549525 4427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:58503
I20250813 07:54:12.554008 4614 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:54:12.592265 4506 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:12.592970 4506 tablet_bootstrap.cc:492] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:54:12.594022 4506 ts_tablet_manager.cc:1397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.134s user 0.119s sys 0.012s
I20250813 07:54:12.595577 4506 raft_consensus.cc:357] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:54:12.595952 4506 raft_consensus.cc:738] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: FOLLOWER
I20250813 07:54:12.596432 4506 consensus_queue.cc:260] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:54:12.596834 4506 raft_consensus.cc:397] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:12.597095 4506 raft_consensus.cc:491] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:12.597406 4506 raft_consensus.cc:3058] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:54:12.602933 4506 raft_consensus.cc:513] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:54:12.603675 4506 leader_election.cc:304] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7eeb2a92c2447f19dc82b6ddcf47232; no voters:
I20250813 07:54:12.604184 4506 leader_election.cc:290] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250813 07:54:12.604414 4618 raft_consensus.cc:2802] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 07:54:12.612382 4618 raft_consensus.cc:695] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 LEADER]: Becoming Leader. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Running, Role: LEADER
I20250813 07:54:12.613459 4618 consensus_queue.cc:237] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } }
I20250813 07:54:12.620631 4506 ts_tablet_manager.cc:1428] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.026s user 0.018s sys 0.008s
I20250813 07:54:12.621567 4506 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap starting.
I20250813 07:54:12.633630 4426 catalog_manager.cc:5582] T 2e81f468bea34fe0a5baee429c75ddb4 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } health_report { overall_health: HEALTHY } } }
I20250813 07:54:12.736188 4506 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:12.737072 4506 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Bootstrap complete.
I20250813 07:54:12.738576 4506 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent bootstrapping tablet: real 0.117s user 0.093s sys 0.020s
I20250813 07:54:12.740594 4506 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:12.741233 4506 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Initialized, Role: FOLLOWER
I20250813 07:54:12.741888 4506 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:12.743979 4506 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20250813 07:54:12.831446 4619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:12.831931 4619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:12.832415 4619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:12.864665 4619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:12.865545 4619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:54:12.900523 4619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:41055
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=41813
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:12.901955 4619 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:12.903538 4619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:12.915602 4635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:13.836742 4639 raft_consensus.cc:491] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:54:13.837229 4639 raft_consensus.cc:513] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:13.839838 4639 leader_election.cc:290] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557), 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
W20250813 07:54:13.845170 4502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
W20250813 07:54:13.853886 4502 leader_election.cc:336] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
W20250813 07:54:13.854578 4502 leader_election.cc:336] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055): Network error: Client connection negotiation failed: client connection to 127.0.106.194:41055: connect: Connection refused (error 111)
I20250813 07:54:13.854967 4502 leader_election.cc:304] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7eeb2a92c2447f19dc82b6ddcf47232; no voters: 1f4b42a1a37b4f2b845b0685441c0091, 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:13.855700 4639 raft_consensus.cc:2747] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250813 07:54:12.916478 4636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:14.033018 4619 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.119s user 0.404s sys 0.712s
W20250813 07:54:14.033401 4619 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.119s user 0.405s sys 0.713s
I20250813 07:54:14.035156 4619 server_base.cc:1047] running on GCE node
W20250813 07:54:14.036835 4644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:14.038019 4619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:14.040159 4619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:14.041534 4619 hybrid_clock.cc:648] HybridClock initialized: now 1755071654041503 us; error 37 us; skew 500 ppm
I20250813 07:54:14.042366 4619 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:14.049305 4619 webserver.cc:480] Webserver started at http://127.0.106.194:41813/ using document root <none> and password file <none>
I20250813 07:54:14.050264 4619 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:14.050486 4619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:14.059044 4619 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250813 07:54:14.064246 4649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:14.065377 4619 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250813 07:54:14.065721 4619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1f4b42a1a37b4f2b845b0685441c0091"
format_stamp: "Formatted at 2025-08-13 07:53:45 on dist-test-slave-dr1z"
I20250813 07:54:14.067660 4619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:14.139072 4619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:14.140591 4619 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:14.141034 4619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:14.143625 4619 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:14.149134 4656 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250813 07:54:14.160329 4619 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250813 07:54:14.160579 4619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250813 07:54:14.160856 4619 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250813 07:54:14.165906 4656 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap starting.
I20250813 07:54:14.168540 4619 ts_tablet_manager.cc:610] Registered 2 tablets
I20250813 07:54:14.168761 4619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.005s sys 0.000s
I20250813 07:54:14.219735 4656 log.cc:826] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:14.328306 4619 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:41055
I20250813 07:54:14.328470 4763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:41055 every 8 connection(s)
I20250813 07:54:14.331799 4619 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:54:14.334352 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4619
I20250813 07:54:14.336165 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:46557
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:37727
--webserver_port=45785
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 07:54:14.356518 4656 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:14.357758 4656 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap complete.
I20250813 07:54:14.359586 4656 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent bootstrapping tablet: real 0.194s user 0.140s sys 0.049s
I20250813 07:54:14.378365 4656 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:14.380478 4656 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Initialized, Role: FOLLOWER
I20250813 07:54:14.381423 4656 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:14.385071 4656 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent starting tablet: real 0.025s user 0.015s sys 0.004s
I20250813 07:54:14.385803 4656 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap starting.
I20250813 07:54:14.386853 4764 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:54:14.387331 4764 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:14.388754 4764 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:54:14.393335 4426 ts_manager.cc:194] Registered new tserver with Master: 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:54:14.397414 4426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:55045
I20250813 07:54:14.401916 4764 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:54:14.475683 4656 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:14.476323 4656 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Bootstrap complete.
I20250813 07:54:14.477332 4656 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent bootstrapping tablet: real 0.092s user 0.078s sys 0.009s
I20250813 07:54:14.478847 4656 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:14.479279 4656 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4b42a1a37b4f2b845b0685441c0091, State: Initialized, Role: FOLLOWER
I20250813 07:54:14.479769 4656 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:14.481019 4656 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250813 07:54:14.521976 4771 raft_consensus.cc:491] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:54:14.522400 4771 raft_consensus.cc:513] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:14.524151 4771 leader_election.cc:290] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055), 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
W20250813 07:54:14.531054 4502 leader_election.cc:336] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
I20250813 07:54:14.538036 4719 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf4d758045124a889faf25713edb5904" candidate_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "1f4b42a1a37b4f2b845b0685441c0091" is_pre_election: true
I20250813 07:54:14.538736 4719 raft_consensus.cc:2466] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7eeb2a92c2447f19dc82b6ddcf47232 in term 2.
I20250813 07:54:14.539983 4502 leader_election.cc:304] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091, b7eeb2a92c2447f19dc82b6ddcf47232; no voters: 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:14.540762 4771 raft_consensus.cc:2802] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250813 07:54:14.541041 4771 raft_consensus.cc:491] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:54:14.541313 4771 raft_consensus.cc:3058] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:54:14.545783 4771 raft_consensus.cc:513] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:14.547333 4771 leader_election.cc:290] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Requested vote from peers 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055), 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:54:14.548148 4719 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf4d758045124a889faf25713edb5904" candidate_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "1f4b42a1a37b4f2b845b0685441c0091"
I20250813 07:54:14.548671 4719 raft_consensus.cc:3058] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Advancing to term 3
W20250813 07:54:14.551363 4502 leader_election.cc:336] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
I20250813 07:54:14.555456 4719 raft_consensus.cc:2466] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7eeb2a92c2447f19dc82b6ddcf47232 in term 3.
I20250813 07:54:14.556303 4502 leader_election.cc:304] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091, b7eeb2a92c2447f19dc82b6ddcf47232; no voters: 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:14.557045 4771 raft_consensus.cc:2802] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 07:54:14.558637 4771 raft_consensus.cc:695] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 LEADER]: Becoming Leader. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Running, Role: LEADER
I20250813 07:54:14.559397 4771 consensus_queue.cc:237] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:14.566897 4426 catalog_manager.cc:5582] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: term changed from 2 to 3, leader changed from 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) to b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193). New cstate: current_term: 3 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250813 07:54:14.672434 4768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:14.672926 4768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:14.673404 4768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:14.705833 4768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:14.706676 4768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:54:14.741575 4768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:46557
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=45785
--tserver_master_addrs=127.0.106.254:37727
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:14.742838 4768 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:14.744619 4768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:14.755975 4787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:15.021091 4719 raft_consensus.cc:1273] T bf4d758045124a889faf25713edb5904 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Refusing update from remote peer b7eeb2a92c2447f19dc82b6ddcf47232: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250813 07:54:15.023545 4771 consensus_queue.cc:1035] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20250813 07:54:15.112461 4502 consensus_peers.cc:489] T bf4d758045124a889faf25713edb5904 P b7eeb2a92c2447f19dc82b6ddcf47232 -> Peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Couldn't send request to peer 8d97630ce3164951a12fe25d95ec0dd0. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:15.359417 4771 raft_consensus.cc:491] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:54:15.360020 4771 raft_consensus.cc:513] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:15.362437 4771 leader_election.cc:290] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557), 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:54:15.366381 4718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6435994656e46f7a213a7f754bfc907" candidate_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "1f4b42a1a37b4f2b845b0685441c0091" is_pre_election: true
I20250813 07:54:15.367120 4718 raft_consensus.cc:2466] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7eeb2a92c2447f19dc82b6ddcf47232 in term 2.
I20250813 07:54:15.369128 4502 leader_election.cc:304] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091, b7eeb2a92c2447f19dc82b6ddcf47232; no voters:
I20250813 07:54:15.374449 4771 raft_consensus.cc:2802] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250813 07:54:15.374930 4771 raft_consensus.cc:491] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:54:15.375859 4771 raft_consensus.cc:3058] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 2 FOLLOWER]: Advancing to term 3
W20250813 07:54:15.375432 4502 leader_election.cc:336] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
I20250813 07:54:15.392153 4771 raft_consensus.cc:513] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
W20250813 07:54:15.400195 4502 leader_election.cc:336] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111)
I20250813 07:54:15.403132 4718 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c6435994656e46f7a213a7f754bfc907" candidate_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "1f4b42a1a37b4f2b845b0685441c0091"
I20250813 07:54:15.403769 4718 raft_consensus.cc:3058] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:54:15.411902 4718 raft_consensus.cc:2466] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7eeb2a92c2447f19dc82b6ddcf47232 in term 3.
I20250813 07:54:15.414084 4502 leader_election.cc:304] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f4b42a1a37b4f2b845b0685441c0091, b7eeb2a92c2447f19dc82b6ddcf47232; no voters: 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:15.415491 4776 raft_consensus.cc:2802] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 07:54:15.412642 4771 leader_election.cc:290] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [CANDIDATE]: Term 3 election: Requested vote from peers 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557), 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055)
I20250813 07:54:15.427177 4776 raft_consensus.cc:695] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 LEADER]: Becoming Leader. State: Replica: b7eeb2a92c2447f19dc82b6ddcf47232, State: Running, Role: LEADER
I20250813 07:54:15.428313 4776 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:15.451124 4426 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: term changed from 2 to 3, leader changed from <none> to b7eeb2a92c2447f19dc82b6ddcf47232 (127.0.106.193). New cstate: current_term: 3 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 07:54:15.813129 4718 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Refusing update from remote peer b7eeb2a92c2447f19dc82b6ddcf47232: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250813 07:54:15.815092 4776 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20250813 07:54:15.886292 4502 consensus_peers.cc:489] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 -> Peer 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): Couldn't send request to peer 8d97630ce3164951a12fe25d95ec0dd0. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.195:46557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:15.926060 4569 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:15.932472 4719 raft_consensus.cc:1273] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Refusing update from remote peer b7eeb2a92c2447f19dc82b6ddcf47232: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250813 07:54:15.934080 4776 consensus_queue.cc:1035] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20250813 07:54:15.943428 4799 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:15.951504 4718 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:15.969746 4426 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 reported cstate change: config changed from index 13 to 15, VOTER 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195) evicted. New cstate: current_term: 3 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:15.977072 4414 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c6435994656e46f7a213a7f754bfc907 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20250813 07:54:15.982753 4426 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet c6435994656e46f7a213a7f754bfc907 on TS 8d97630ce3164951a12fe25d95ec0dd0: Not found: failed to reset TS proxy: Could not find TS for UUID 8d97630ce3164951a12fe25d95ec0dd0
W20250813 07:54:15.985256 4789 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1224 milliseconds
W20250813 07:54:15.985649 4790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:15.987380 4768 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.230s user 0.452s sys 0.704s
I20250813 07:54:15.987130 4569 consensus_queue.cc:237] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
W20250813 07:54:15.987759 4768 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.231s user 0.454s sys 0.705s
I20250813 07:54:15.988026 4768 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 07:54:14.757324 4788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:15.991146 4799 raft_consensus.cc:2953] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } }
I20250813 07:54:15.991993 4768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:15.998708 4414 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet c6435994656e46f7a213a7f754bfc907 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 07:54:15.999926 4768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:16.001526 4768 hybrid_clock.cc:648] HybridClock initialized: now 1755071656001451 us; error 54 us; skew 500 ppm
I20250813 07:54:16.002681 4768 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:16.002887 4427 catalog_manager.cc:5582] T c6435994656e46f7a213a7f754bfc907 P b7eeb2a92c2447f19dc82b6ddcf47232 reported cstate change: config changed from index 15 to 16, VOTER 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194) evicted. New cstate: current_term: 3 leader_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 07:54:16.011384 4768 webserver.cc:480] Webserver started at http://127.0.106.195:45785/ using document root <none> and password file <none>
I20250813 07:54:16.012719 4768 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:16.013084 4768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:16.023268 4699 tablet_service.cc:1515] Processing DeleteTablet for tablet c6435994656e46f7a213a7f754bfc907 with delete_type TABLET_DATA_TOMBSTONED (TS 1f4b42a1a37b4f2b845b0685441c0091 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:59160
I20250813 07:54:16.023838 4768 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.002s
I20250813 07:54:16.026572 4815 tablet_replica.cc:331] stopping tablet replica
I20250813 07:54:16.027400 4815 raft_consensus.cc:2241] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250813 07:54:16.028105 4815 raft_consensus.cc:2270] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250813 07:54:16.029842 4816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:16.030933 4768 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250813 07:54:16.031895 4815 ts_tablet_manager.cc:1905] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 07:54:16.031258 4768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8d97630ce3164951a12fe25d95ec0dd0"
format_stamp: "Formatted at 2025-08-13 07:53:47 on dist-test-slave-dr1z"
I20250813 07:54:16.033245 4768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250813 07:54:16.040232 4412 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet c6435994656e46f7a213a7f754bfc907 on TS 8d97630ce3164951a12fe25d95ec0dd0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8d97630ce3164951a12fe25d95ec0dd0
I20250813 07:54:16.047283 4815 ts_tablet_manager.cc:1918] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250813 07:54:16.047708 4815 log.cc:1199] T c6435994656e46f7a213a7f754bfc907 P 1f4b42a1a37b4f2b845b0685441c0091: Deleting WAL directory at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/wals/c6435994656e46f7a213a7f754bfc907
I20250813 07:54:16.049808 4413 catalog_manager.cc:4928] TS 1f4b42a1a37b4f2b845b0685441c0091 (127.0.106.194:41055): tablet c6435994656e46f7a213a7f754bfc907 (table TestTable [id=2320e9785f604267b57e056dfe72d135]) successfully deleted
I20250813 07:54:16.098022 4768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:16.100016 4768 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:16.100698 4768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:16.104089 4768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:16.109530 4823 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250813 07:54:16.120886 4768 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250813 07:54:16.121133 4768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250813 07:54:16.121366 4768 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250813 07:54:16.126672 4823 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap starting.
I20250813 07:54:16.129125 4768 ts_tablet_manager.cc:610] Registered 2 tablets
I20250813 07:54:16.129328 4768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.002s sys 0.002s
I20250813 07:54:16.192471 4823 log.cc:826] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:16.293973 4823 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:16.294728 4768 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:46557
I20250813 07:54:16.294992 4930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:46557 every 8 connection(s)
I20250813 07:54:16.295179 4823 tablet_bootstrap.cc:492] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap complete.
I20250813 07:54:16.296887 4823 ts_tablet_manager.cc:1397] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent bootstrapping tablet: real 0.171s user 0.139s sys 0.028s
I20250813 07:54:16.297348 4768 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:54:16.305940 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4768
I20250813 07:54:16.318495 4823 raft_consensus.cc:357] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:16.321704 4823 raft_consensus.cc:738] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Initialized, Role: FOLLOWER
I20250813 07:54:16.322769 4823 consensus_queue.cc:260] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } } peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } attrs { promote: false } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } }
I20250813 07:54:16.367903 4823 ts_tablet_manager.cc:1428] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent starting tablet: real 0.071s user 0.054s sys 0.016s
I20250813 07:54:16.368809 4823 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap starting.
I20250813 07:54:16.376932 4931 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37727
I20250813 07:54:16.377362 4931 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:16.378350 4931 heartbeater.cc:507] Master 127.0.106.254:37727 requested a full tablet report, sending...
I20250813 07:54:16.382718 4426 ts_manager.cc:194] Registered new tserver with Master: 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557)
I20250813 07:54:16.387105 4426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:59451
I20250813 07:54:16.392333 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:54:16.393904 4931 heartbeater.cc:499] Master 127.0.106.254:37727 was elected leader, sending a full tablet report...
I20250813 07:54:16.397799 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250813 07:54:16.401639 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250813 07:54:16.466378 4823 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:54:16.467006 4823 tablet_bootstrap.cc:492] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Bootstrap complete.
I20250813 07:54:16.467928 4823 ts_tablet_manager.cc:1397] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent bootstrapping tablet: real 0.099s user 0.088s sys 0.008s
I20250813 07:54:16.469458 4823 raft_consensus.cc:357] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:16.470103 4823 raft_consensus.cc:738] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d97630ce3164951a12fe25d95ec0dd0, State: Initialized, Role: FOLLOWER
I20250813 07:54:16.470690 4823 consensus_queue.cc:260] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1f4b42a1a37b4f2b845b0685441c0091" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 41055 } } peers { permanent_uuid: "b7eeb2a92c2447f19dc82b6ddcf47232" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 41401 } attrs { promote: false } } peers { permanent_uuid: "8d97630ce3164951a12fe25d95ec0dd0" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 46557 } attrs { promote: false } }
I20250813 07:54:16.471884 4823 ts_tablet_manager.cc:1428] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250813 07:54:16.573844 4886 raft_consensus.cc:3058] T bf4d758045124a889faf25713edb5904 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Advancing to term 3
I20250813 07:54:16.661039 4866 tablet_service.cc:1515] Processing DeleteTablet for tablet c6435994656e46f7a213a7f754bfc907 with delete_type TABLET_DATA_TOMBSTONED (TS 8d97630ce3164951a12fe25d95ec0dd0 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:55962
I20250813 07:54:16.665794 4943 tablet_replica.cc:331] stopping tablet replica
I20250813 07:54:16.666438 4943 raft_consensus.cc:2241] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 07:54:16.666860 4943 raft_consensus.cc:2270] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 07:54:16.669209 4943 ts_tablet_manager.cc:1905] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 07:54:16.679011 4943 ts_tablet_manager.cc:1918] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250813 07:54:16.679337 4943 log.cc:1199] T c6435994656e46f7a213a7f754bfc907 P 8d97630ce3164951a12fe25d95ec0dd0: Deleting WAL directory at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/wals/c6435994656e46f7a213a7f754bfc907
I20250813 07:54:16.680670 4413 catalog_manager.cc:4928] TS 8d97630ce3164951a12fe25d95ec0dd0 (127.0.106.195:46557): tablet c6435994656e46f7a213a7f754bfc907 (table TestTable [id=2320e9785f604267b57e056dfe72d135]) successfully deleted
W20250813 07:54:17.405791 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:18.409396 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:19.413055 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:20.416326 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:21.420131 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:22.423784 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:23.427416 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:24.431023 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:25.434372 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:26.437414 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:27.441448 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:28.445196 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:29.448858 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:30.452361 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:31.455996 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:32.459465 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:33.462954 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:34.466676 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 07:54:35.128165 4610 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5060 after lost signal to thread 4484
W20250813 07:54:35.129057 4610 debug-util.cc:398] Leaking SignalData structure 0x7b08000b50c0 after lost signal to thread 4613
W20250813 07:54:35.470082 427 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet c6435994656e46f7a213a7f754bfc907: tablet_id: "c6435994656e46f7a213a7f754bfc907" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250813 07:54:36.473888 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4464
I20250813 07:54:36.498004 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4619
I20250813 07:54:36.519419 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4768
I20250813 07:54:36.544342 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4394
2025-08-13T07:54:36Z chronyd exiting
I20250813 07:54:36.588745 427 test_util.cc:183] -----------------------------------------------
I20250813 07:54:36.588956 427 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755071556474372-427-0
[ FAILED ] AdminCliTest.TestRebuildTables (56367 ms)
[----------] 5 tests from AdminCliTest (120044 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250813 07:54:36.592552 427 test_util.cc:276] Using random seed: 2010558862
I20250813 07:54:36.596773 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:54:36.596944 427 ts_itest-base.cc:116] --------------
I20250813 07:54:36.597061 427 ts_itest-base.cc:117] 5 tablet servers
I20250813 07:54:36.597167 427 ts_itest-base.cc:118] 3 replicas per TS
I20250813 07:54:36.597270 427 ts_itest-base.cc:119] --------------
2025-08-13T07:54:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:54:36Z Disabled control of system clock
I20250813 07:54:36.638710 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37093
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:37093
--raft_prepare_replacement_before_eviction=true with env {}
W20250813 07:54:36.936789 4965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:36.937321 4965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:36.937783 4965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:36.968987 4965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:36.969333 4965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:54:36.969568 4965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:36.969812 4965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:54:36.970013 4965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:54:37.005221 4965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:37093
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:37.006445 4965 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:37.008009 4965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:37.018810 4972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:37.019061 4973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:38.203017 4975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:38.205116 4974 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1181 milliseconds
W20250813 07:54:38.206130 4965 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.188s user 0.430s sys 0.749s
W20250813 07:54:38.206382 4965 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.188s user 0.430s sys 0.749s
I20250813 07:54:38.206604 4965 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:38.207769 4965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:38.210494 4965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:38.211875 4965 hybrid_clock.cc:648] HybridClock initialized: now 1755071678211848 us; error 43 us; skew 500 ppm
I20250813 07:54:38.212661 4965 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:38.220146 4965 webserver.cc:480] Webserver started at http://127.0.106.254:39183/ using document root <none> and password file <none>
I20250813 07:54:38.221097 4965 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:38.221319 4965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:38.221788 4965 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:38.226264 4965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6b782816a93e4bb9b2112e1f8595f01d"
format_stamp: "Formatted at 2025-08-13 07:54:38 on dist-test-slave-dr1z"
I20250813 07:54:38.227381 4965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6b782816a93e4bb9b2112e1f8595f01d"
format_stamp: "Formatted at 2025-08-13 07:54:38 on dist-test-slave-dr1z"
I20250813 07:54:38.235211 4965 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250813 07:54:38.241067 4982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:38.242323 4965 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250813 07:54:38.242679 4965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6b782816a93e4bb9b2112e1f8595f01d"
format_stamp: "Formatted at 2025-08-13 07:54:38 on dist-test-slave-dr1z"
I20250813 07:54:38.243026 4965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:38.312587 4965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:38.314172 4965 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:38.314591 4965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:38.380252 4965 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:37093
I20250813 07:54:38.380326 5033 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:37093 every 8 connection(s)
I20250813 07:54:38.382938 4965 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:54:38.387838 5034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:38.389654 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 4965
I20250813 07:54:38.390349 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:54:38.408953 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d: Bootstrap starting.
I20250813 07:54:38.414253 5034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d: Neither blocks nor log segments found. Creating new log.
I20250813 07:54:38.416296 5034 log.cc:826] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:38.420825 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d: No bootstrap required, opened a new log
I20250813 07:54:38.439431 5034 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } }
I20250813 07:54:38.440083 5034 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:54:38.440289 5034 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b782816a93e4bb9b2112e1f8595f01d, State: Initialized, Role: FOLLOWER
I20250813 07:54:38.440891 5034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } }
I20250813 07:54:38.441366 5034 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:38.441628 5034 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:38.441895 5034 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:54:38.445866 5034 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } }
I20250813 07:54:38.446586 5034 leader_election.cc:304] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b782816a93e4bb9b2112e1f8595f01d; no voters:
I20250813 07:54:38.448158 5034 leader_election.cc:290] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:54:38.448868 5039 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:54:38.450809 5039 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [term 1 LEADER]: Becoming Leader. State: Replica: 6b782816a93e4bb9b2112e1f8595f01d, State: Running, Role: LEADER
I20250813 07:54:38.451510 5039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } }
I20250813 07:54:38.452471 5034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:54:38.458102 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b782816a93e4bb9b2112e1f8595f01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } } }
I20250813 07:54:38.458668 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b782816a93e4bb9b2112e1f8595f01d. Latest consensus state: current_term: 1 leader_uuid: "6b782816a93e4bb9b2112e1f8595f01d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b782816a93e4bb9b2112e1f8595f01d" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 37093 } } }
I20250813 07:54:38.458947 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:38.459262 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:38.463500 5047 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:54:38.473737 5047 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:54:38.489746 5047 catalog_manager.cc:1349] Generated new cluster ID: 29b28aab8e0d4b01ab71317daec2437a
I20250813 07:54:38.490078 5047 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:54:38.506136 5047 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:54:38.507768 5047 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:54:38.523514 5047 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6b782816a93e4bb9b2112e1f8595f01d: Generated new TSK 0
I20250813 07:54:38.524565 5047 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:54:38.546325 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250813 07:54:38.849107 5058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:38.849639 5058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:38.850127 5058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:38.881562 5058 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:38.881973 5058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:38.882766 5058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:54:38.917796 5058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:38.919057 5058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:38.920701 5058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:38.932945 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:40.334685 5063 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 5058
W20250813 07:54:40.561761 5063 kernel_stack_watchdog.cc:198] Thread 5058 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 07:54:38.933976 5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:40.562866 5058 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.629s user 0.620s sys 0.940s
W20250813 07:54:40.563066 5066 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1629 milliseconds
W20250813 07:54:40.563400 5058 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.630s user 0.620s sys 0.941s
I20250813 07:54:40.564239 5058 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 07:54:40.564327 5067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:40.568289 5058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:40.570645 5058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:40.572057 5058 hybrid_clock.cc:648] HybridClock initialized: now 1755071680572010 us; error 62 us; skew 500 ppm
I20250813 07:54:40.572957 5058 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:40.579620 5058 webserver.cc:480] Webserver started at http://127.0.106.193:43455/ using document root <none> and password file <none>
I20250813 07:54:40.580636 5058 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:40.580880 5058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:40.581334 5058 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:40.586351 5058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d"
format_stamp: "Formatted at 2025-08-13 07:54:40 on dist-test-slave-dr1z"
I20250813 07:54:40.587534 5058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d"
format_stamp: "Formatted at 2025-08-13 07:54:40 on dist-test-slave-dr1z"
I20250813 07:54:40.595541 5058 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250813 07:54:40.601923 5074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:40.603327 5058 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250813 07:54:40.603674 5058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d"
format_stamp: "Formatted at 2025-08-13 07:54:40 on dist-test-slave-dr1z"
I20250813 07:54:40.604038 5058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:40.655215 5058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:40.656922 5058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:40.657398 5058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:40.660527 5058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:40.665028 5058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:54:40.665380 5058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:40.665673 5058 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:54:40.665841 5058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:40.827594 5058 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:44853
I20250813 07:54:40.827685 5186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:44853 every 8 connection(s)
I20250813 07:54:40.830389 5058 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:54:40.831527 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5058
I20250813 07:54:40.831993 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:54:40.841884 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 07:54:40.853119 5187 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37093
I20250813 07:54:40.853662 5187 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:40.854939 5187 heartbeater.cc:507] Master 127.0.106.254:37093 requested a full tablet report, sending...
I20250813 07:54:40.858285 4999 ts_manager.cc:194] Registered new tserver with Master: c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853)
I20250813 07:54:40.861344 4999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:45519
W20250813 07:54:41.148423 5191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:41.148991 5191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:41.149526 5191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:41.181247 5191 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:41.181691 5191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:41.182482 5191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:54:41.218045 5191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:41.219331 5191 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:41.221006 5191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:41.232640 5197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:41.866206 5187 heartbeater.cc:499] Master 127.0.106.254:37093 was elected leader, sending a full tablet report...
W20250813 07:54:41.234125 5198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:42.587735 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:42.591058 5199 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1353 milliseconds
W20250813 07:54:42.591552 5191 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.358s user 0.546s sys 0.798s
W20250813 07:54:42.591775 5191 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.359s user 0.546s sys 0.799s
I20250813 07:54:42.591992 5191 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:42.593231 5191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:42.595526 5191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:42.596899 5191 hybrid_clock.cc:648] HybridClock initialized: now 1755071682596869 us; error 43 us; skew 500 ppm
I20250813 07:54:42.597796 5191 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:42.605367 5191 webserver.cc:480] Webserver started at http://127.0.106.194:36137/ using document root <none> and password file <none>
I20250813 07:54:42.606331 5191 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:42.606563 5191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:42.607004 5191 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:42.611750 5191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "16a890036f304ea8868d1f44312bd25d"
format_stamp: "Formatted at 2025-08-13 07:54:42 on dist-test-slave-dr1z"
I20250813 07:54:42.612867 5191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "16a890036f304ea8868d1f44312bd25d"
format_stamp: "Formatted at 2025-08-13 07:54:42 on dist-test-slave-dr1z"
I20250813 07:54:42.620777 5191 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250813 07:54:42.627113 5207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:42.628427 5191 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250813 07:54:42.628749 5191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "16a890036f304ea8868d1f44312bd25d"
format_stamp: "Formatted at 2025-08-13 07:54:42 on dist-test-slave-dr1z"
I20250813 07:54:42.629091 5191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:42.701678 5191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:42.703138 5191 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:42.703585 5191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:42.706059 5191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:42.710109 5191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:54:42.710316 5191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:42.710557 5191 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:54:42.710712 5191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:42.839718 5191 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:34195
I20250813 07:54:42.839831 5319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:34195 every 8 connection(s)
I20250813 07:54:42.842211 5191 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 07:54:42.844007 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5191
I20250813 07:54:42.844512 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 07:54:42.851648 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 07:54:42.863201 5320 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37093
I20250813 07:54:42.863620 5320 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:42.864601 5320 heartbeater.cc:507] Master 127.0.106.254:37093 requested a full tablet report, sending...
I20250813 07:54:42.866698 4999 ts_manager.cc:194] Registered new tserver with Master: 16a890036f304ea8868d1f44312bd25d (127.0.106.194:34195)
I20250813 07:54:42.867904 4999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:54375
W20250813 07:54:43.147719 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:43.148208 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:43.148698 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:43.180132 5324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:43.180549 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:43.181334 5324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:54:43.216269 5324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:43.217620 5324 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:43.219172 5324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:43.230476 5330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:43.870895 5320 heartbeater.cc:499] Master 127.0.106.254:37093 was elected leader, sending a full tablet report...
W20250813 07:54:43.231038 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:43.234460 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:44.316172 5332 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250813 07:54:44.316272 5324 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:44.319865 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:44.321998 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:44.323369 5324 hybrid_clock.cc:648] HybridClock initialized: now 1755071684323341 us; error 39 us; skew 500 ppm
I20250813 07:54:44.324179 5324 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:44.330614 5324 webserver.cc:480] Webserver started at http://127.0.106.195:44885/ using document root <none> and password file <none>
I20250813 07:54:44.331522 5324 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:44.331732 5324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:44.332190 5324 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:44.336433 5324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "27bda42fef8b4df98914c8e57b06bb89"
format_stamp: "Formatted at 2025-08-13 07:54:44 on dist-test-slave-dr1z"
I20250813 07:54:44.337659 5324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "27bda42fef8b4df98914c8e57b06bb89"
format_stamp: "Formatted at 2025-08-13 07:54:44 on dist-test-slave-dr1z"
I20250813 07:54:44.344556 5324 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250813 07:54:44.350322 5340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:44.351518 5324 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:54:44.351915 5324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "27bda42fef8b4df98914c8e57b06bb89"
format_stamp: "Formatted at 2025-08-13 07:54:44 on dist-test-slave-dr1z"
I20250813 07:54:44.352272 5324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:44.399219 5324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:44.400705 5324 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:44.401139 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:44.403738 5324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:44.407805 5324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:54:44.408021 5324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:44.408265 5324 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:54:44.408419 5324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:44.556660 5324 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:36439
I20250813 07:54:44.556761 5452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:36439 every 8 connection(s)
I20250813 07:54:44.559310 5324 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 07:54:44.563043 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5324
I20250813 07:54:44.563520 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 07:54:44.570569 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.196:0
--local_ip_for_outbound_sockets=127.0.106.196
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 07:54:44.584151 5453 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37093
I20250813 07:54:44.584571 5453 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:44.585667 5453 heartbeater.cc:507] Master 127.0.106.254:37093 requested a full tablet report, sending...
I20250813 07:54:44.587636 4999 ts_manager.cc:194] Registered new tserver with Master: 27bda42fef8b4df98914c8e57b06bb89 (127.0.106.195:36439)
I20250813 07:54:44.588850 4999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:57887
W20250813 07:54:44.863034 5457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:44.863513 5457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:44.864002 5457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:44.895550 5457 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:44.895942 5457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:44.896735 5457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.196
I20250813 07:54:44.931681 5457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.106.196
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.196
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:44.933107 5457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:44.934873 5457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:44.948644 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:45.550894 5030 debug-util.cc:398] Leaking SignalData structure 0x7b080007e1a0 after lost signal to thread 4967
I20250813 07:54:45.591881 5453 heartbeater.cc:499] Master 127.0.106.254:37093 was elected leader, sending a full tablet report...
W20250813 07:54:44.948900 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:46.172876 5465 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1223 milliseconds
W20250813 07:54:46.173959 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:46.174101 5457 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:46.178040 5457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:46.184005 5457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:46.185427 5457 hybrid_clock.cc:648] HybridClock initialized: now 1755071686185364 us; error 75 us; skew 500 ppm
I20250813 07:54:46.186293 5457 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:46.192749 5457 webserver.cc:480] Webserver started at http://127.0.106.196:39381/ using document root <none> and password file <none>
I20250813 07:54:46.193691 5457 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:46.193921 5457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:46.194408 5457 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:46.198832 5457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1c16f21611464c3692d20959f52afb44"
format_stamp: "Formatted at 2025-08-13 07:54:46 on dist-test-slave-dr1z"
I20250813 07:54:46.199899 5457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1c16f21611464c3692d20959f52afb44"
format_stamp: "Formatted at 2025-08-13 07:54:46 on dist-test-slave-dr1z"
I20250813 07:54:46.206954 5457 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250813 07:54:46.212491 5473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:46.213519 5457 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:54:46.213795 5457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1c16f21611464c3692d20959f52afb44"
format_stamp: "Formatted at 2025-08-13 07:54:46 on dist-test-slave-dr1z"
I20250813 07:54:46.214099 5457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:46.262683 5457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:46.264117 5457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:46.264525 5457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:46.267383 5457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:46.271739 5457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:54:46.271936 5457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:46.272188 5457 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:54:46.272336 5457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:46.426748 5457 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.196:46881
I20250813 07:54:46.426920 5585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.196:46881 every 8 connection(s)
I20250813 07:54:46.429240 5457 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250813 07:54:46.432801 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5457
I20250813 07:54:46.433302 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250813 07:54:46.441318 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.197:0
--local_ip_for_outbound_sockets=127.0.106.197
--webserver_interface=127.0.106.197
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--builtin_ntp_servers=127.0.106.212:43597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 07:54:46.472908 5586 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37093
I20250813 07:54:46.473344 5586 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:46.474372 5586 heartbeater.cc:507] Master 127.0.106.254:37093 requested a full tablet report, sending...
I20250813 07:54:46.476514 4999 ts_manager.cc:194] Registered new tserver with Master: 1c16f21611464c3692d20959f52afb44 (127.0.106.196:46881)
I20250813 07:54:46.477762 4999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.196:60315
W20250813 07:54:46.798198 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:46.798676 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:46.799157 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:46.838388 5590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 07:54:46.838771 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:46.839545 5590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.197
I20250813 07:54:46.874930 5590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:43597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.106.197
--webserver_port=0
--tserver_master_addrs=127.0.106.254:37093
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.197
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:46.876224 5590 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:46.877820 5590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:46.890404 5597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:54:47.480702 5586 heartbeater.cc:499] Master 127.0.106.254:37093 was elected leader, sending a full tablet report...
W20250813 07:54:46.891602 5598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:48.132267 5600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:48.134073 5590 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.242s user 0.002s sys 0.007s
W20250813 07:54:48.134429 5590 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.242s user 0.002s sys 0.007s
I20250813 07:54:48.134688 5590 server_base.cc:1047] running on GCE node
I20250813 07:54:48.136022 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:48.138648 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:48.140183 5590 hybrid_clock.cc:648] HybridClock initialized: now 1755071688140129 us; error 46 us; skew 500 ppm
I20250813 07:54:48.141229 5590 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:48.149935 5590 webserver.cc:480] Webserver started at http://127.0.106.197:35931/ using document root <none> and password file <none>
I20250813 07:54:48.151288 5590 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:48.151571 5590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:48.152165 5590 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:48.158775 5590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "635d0b56b4884c8e922d86644c5d4be5"
format_stamp: "Formatted at 2025-08-13 07:54:48 on dist-test-slave-dr1z"
I20250813 07:54:48.160275 5590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "635d0b56b4884c8e922d86644c5d4be5"
format_stamp: "Formatted at 2025-08-13 07:54:48 on dist-test-slave-dr1z"
I20250813 07:54:48.169793 5590 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s
I20250813 07:54:48.177516 5608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:48.178750 5590 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:54:48.179140 5590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "635d0b56b4884c8e922d86644c5d4be5"
format_stamp: "Formatted at 2025-08-13 07:54:48 on dist-test-slave-dr1z"
I20250813 07:54:48.179590 5590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:48.271651 5590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:48.273021 5590 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:48.273422 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:48.275883 5590 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:54:48.279943 5590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:54:48.280136 5590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:48.280344 5590 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:54:48.280479 5590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:48.430749 5590 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.197:34913
I20250813 07:54:48.430871 5720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.197:34913 every 8 connection(s)
I20250813 07:54:48.434636 5590 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250813 07:54:48.443935 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5590
I20250813 07:54:48.444659 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755071556474372-427-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250813 07:54:48.460875 5721 heartbeater.cc:344] Connected to a master server at 127.0.106.254:37093
I20250813 07:54:48.461293 5721 heartbeater.cc:461] Registering TS with master...
I20250813 07:54:48.462322 5721 heartbeater.cc:507] Master 127.0.106.254:37093 requested a full tablet report, sending...
I20250813 07:54:48.465613 4999 ts_manager.cc:194] Registered new tserver with Master: 635d0b56b4884c8e922d86644c5d4be5 (127.0.106.197:34913)
I20250813 07:54:48.467319 4999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.197:42399
I20250813 07:54:48.468791 427 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250813 07:54:48.509021 4999 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49208:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 07:54:48.589805 5122 tablet_service.cc:1468] Processing CreateTablet for tablet 0edad7874b8d4607b0ef739b9efe0288 (DEFAULT_TABLE table=TestTable [id=0336be79fb8e47f399fd19c2c994b6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:54:48.592350 5122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edad7874b8d4607b0ef739b9efe0288. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:48.595036 5521 tablet_service.cc:1468] Processing CreateTablet for tablet 0edad7874b8d4607b0ef739b9efe0288 (DEFAULT_TABLE table=TestTable [id=0336be79fb8e47f399fd19c2c994b6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:54:48.595793 5656 tablet_service.cc:1468] Processing CreateTablet for tablet 0edad7874b8d4607b0ef739b9efe0288 (DEFAULT_TABLE table=TestTable [id=0336be79fb8e47f399fd19c2c994b6ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:54:48.596895 5521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edad7874b8d4607b0ef739b9efe0288. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:48.597795 5656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edad7874b8d4607b0ef739b9efe0288. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:48.623191 5740 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Bootstrap starting.
I20250813 07:54:48.623215 5741 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Bootstrap starting.
I20250813 07:54:48.627912 5742 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Bootstrap starting.
I20250813 07:54:48.630455 5741 tablet_bootstrap.cc:654] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Neither blocks nor log segments found. Creating new log.
I20250813 07:54:48.632757 5741 log.cc:826] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:48.632846 5740 tablet_bootstrap.cc:654] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Neither blocks nor log segments found. Creating new log.
I20250813 07:54:48.635702 5740 log.cc:826] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:48.635942 5742 tablet_bootstrap.cc:654] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Neither blocks nor log segments found. Creating new log.
I20250813 07:54:48.638298 5741 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: No bootstrap required, opened a new log
I20250813 07:54:48.638661 5742 log.cc:826] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:48.638854 5741 ts_tablet_manager.cc:1397] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250813 07:54:48.641642 5740 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: No bootstrap required, opened a new log
I20250813 07:54:48.643404 5740 ts_tablet_manager.cc:1397] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.015s
I20250813 07:54:48.646282 5742 tablet_bootstrap.cc:492] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: No bootstrap required, opened a new log
I20250813 07:54:48.646776 5742 ts_tablet_manager.cc:1397] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250813 07:54:48.660390 5741 raft_consensus.cc:357] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.661202 5741 raft_consensus.cc:383] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:54:48.661530 5741 raft_consensus.cc:738] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635d0b56b4884c8e922d86644c5d4be5, State: Initialized, Role: FOLLOWER
I20250813 07:54:48.662317 5741 consensus_queue.cc:260] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.665547 5721 heartbeater.cc:499] Master 127.0.106.254:37093 was elected leader, sending a full tablet report...
I20250813 07:54:48.667447 5741 ts_tablet_manager.cc:1428] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Time spent starting tablet: real 0.028s user 0.022s sys 0.006s
I20250813 07:54:48.672904 5742 raft_consensus.cc:357] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.673878 5742 raft_consensus.cc:383] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:54:48.674192 5742 raft_consensus.cc:738] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c16f21611464c3692d20959f52afb44, State: Initialized, Role: FOLLOWER
I20250813 07:54:48.674460 5740 raft_consensus.cc:357] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.675303 5740 raft_consensus.cc:383] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:54:48.675585 5740 raft_consensus.cc:738] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06e2e4cbc5c4ba7abfeab8037b9e23d, State: Initialized, Role: FOLLOWER
I20250813 07:54:48.675056 5742 consensus_queue.cc:260] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.676465 5740 consensus_queue.cc:260] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.679587 5742 ts_tablet_manager.cc:1428] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Time spent starting tablet: real 0.033s user 0.025s sys 0.008s
I20250813 07:54:48.683300 5740 ts_tablet_manager.cc:1428] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Time spent starting tablet: real 0.040s user 0.019s sys 0.017s
W20250813 07:54:48.689695 5722 tablet.cc:2378] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:54:48.703857 5587 tablet.cc:2378] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:54:48.786273 5748 raft_consensus.cc:491] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:54:48.786772 5748 raft_consensus.cc:513] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.789168 5748 leader_election.cc:290] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c16f21611464c3692d20959f52afb44 (127.0.106.196:46881), 635d0b56b4884c8e922d86644c5d4be5 (127.0.106.197:34913)
I20250813 07:54:48.801154 5541 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c16f21611464c3692d20959f52afb44" is_pre_election: true
I20250813 07:54:48.802168 5541 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06e2e4cbc5c4ba7abfeab8037b9e23d in term 0.
I20250813 07:54:48.803269 5076 leader_election.cc:304] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c16f21611464c3692d20959f52afb44, c06e2e4cbc5c4ba7abfeab8037b9e23d; no voters:
I20250813 07:54:48.804021 5748 raft_consensus.cc:2802] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:54:48.804106 5676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d0b56b4884c8e922d86644c5d4be5" is_pre_election: true
I20250813 07:54:48.804329 5748 raft_consensus.cc:491] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:54:48.804574 5748 raft_consensus.cc:3058] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:54:48.804801 5676 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c06e2e4cbc5c4ba7abfeab8037b9e23d in term 0.
I20250813 07:54:48.809405 5748 raft_consensus.cc:513] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.810837 5748 leader_election.cc:290] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [CANDIDATE]: Term 1 election: Requested vote from peers 1c16f21611464c3692d20959f52afb44 (127.0.106.196:46881), 635d0b56b4884c8e922d86644c5d4be5 (127.0.106.197:34913)
I20250813 07:54:48.811693 5541 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c16f21611464c3692d20959f52afb44"
I20250813 07:54:48.811914 5676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d0b56b4884c8e922d86644c5d4be5"
I20250813 07:54:48.812258 5541 raft_consensus.cc:3058] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:54:48.812436 5676 raft_consensus.cc:3058] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:54:48.819696 5676 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06e2e4cbc5c4ba7abfeab8037b9e23d in term 1.
I20250813 07:54:48.819693 5541 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c06e2e4cbc5c4ba7abfeab8037b9e23d in term 1.
I20250813 07:54:48.820752 5076 leader_election.cc:304] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635d0b56b4884c8e922d86644c5d4be5, c06e2e4cbc5c4ba7abfeab8037b9e23d; no voters:
I20250813 07:54:48.821328 5748 raft_consensus.cc:2802] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:54:48.822922 5748 raft_consensus.cc:695] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [term 1 LEADER]: Becoming Leader. State: Replica: c06e2e4cbc5c4ba7abfeab8037b9e23d, State: Running, Role: LEADER
I20250813 07:54:48.824008 5748 consensus_queue.cc:237] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:48.845546 4998 catalog_manager.cc:5582] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d reported cstate change: term changed from 0 to 1, leader changed from <none> to c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193). New cstate: current_term: 1 leader_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } health_report { overall_health: UNKNOWN } } }
W20250813 07:54:48.856006 5188 tablet.cc:2378] T 0edad7874b8d4607b0ef739b9efe0288 P c06e2e4cbc5c4ba7abfeab8037b9e23d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:54:48.891949 427 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250813 07:54:48.895352 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c06e2e4cbc5c4ba7abfeab8037b9e23d to finish bootstrapping
I20250813 07:54:48.908392 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1c16f21611464c3692d20959f52afb44 to finish bootstrapping
I20250813 07:54:48.918990 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 635d0b56b4884c8e922d86644c5d4be5 to finish bootstrapping
I20250813 07:54:48.929296 427 test_util.cc:276] Using random seed: 2022895609
I20250813 07:54:48.954705 427 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250813 07:54:48.955653 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5058
W20250813 07:54:48.978780 5759 negotiation.cc:337] Failed RPC negotiation. Trace:
0813 07:54:48.968130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.106.193:44853 (local address 127.0.0.1:58436)
0813 07:54:48.968857 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0813 07:54:48.968899 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0813 07:54:48.969160 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0813 07:54:48.977232 (+ 8072us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":86}
W20250813 07:54:48.987776 5755 meta_cache.cc:302] tablet 0edad7874b8d4607b0ef739b9efe0288: replica c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853) has failed: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250813 07:54:49.008759 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.026572 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.049767 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.057968 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.080315 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.092805 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.128711 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.146340 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.188582 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.210640 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.253243 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.274824 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.327350 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.354275 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.412372 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.440201 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.509173 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.543298 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.615099 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.649178 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.723582 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.763674 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.847189 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.888566 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:49.978104 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.024258 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.076689 5755 meta_cache.cc:302] tablet 0edad7874b8d4607b0ef739b9efe0288: replica c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853) has failed: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111)
W20250813 07:54:50.121935 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.172350 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.273582 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.325640 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.435091 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.488330 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.603261 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
W20250813 07:54:50.657454 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
I20250813 07:54:50.745918 5781 raft_consensus.cc:491] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:54:50.746300 5781 raft_consensus.cc:513] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:50.748971 5781 leader_election.cc:290] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853), 635d0b56b4884c8e922d86644c5d4be5 (127.0.106.197:34913)
W20250813 07:54:50.754122 5474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111)
W20250813 07:54:50.758004 5474 leader_election.cc:336] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853): Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111)
I20250813 07:54:50.761540 5676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "1c16f21611464c3692d20959f52afb44" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d0b56b4884c8e922d86644c5d4be5" is_pre_election: true
I20250813 07:54:50.762030 5676 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c16f21611464c3692d20959f52afb44 in term 1.
I20250813 07:54:50.763024 5475 leader_election.cc:304] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c16f21611464c3692d20959f52afb44, 635d0b56b4884c8e922d86644c5d4be5; no voters: c06e2e4cbc5c4ba7abfeab8037b9e23d
I20250813 07:54:50.763703 5781 raft_consensus.cc:2802] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250813 07:54:50.764020 5781 raft_consensus.cc:491] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:54:50.764284 5781 raft_consensus.cc:3058] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:54:50.770295 5781 raft_consensus.cc:513] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:50.771843 5781 leader_election.cc:290] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 election: Requested vote from peers c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853), 635d0b56b4884c8e922d86644c5d4be5 (127.0.106.197:34913)
W20250813 07:54:50.773061 5501 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48334: Illegal state: replica 1c16f21611464c3692d20959f52afb44 is not leader of this config: current role FOLLOWER
I20250813 07:54:50.775082 5676 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edad7874b8d4607b0ef739b9efe0288" candidate_uuid: "1c16f21611464c3692d20959f52afb44" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "635d0b56b4884c8e922d86644c5d4be5"
I20250813 07:54:50.775640 5676 raft_consensus.cc:3058] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 1 FOLLOWER]: Advancing to term 2
W20250813 07:54:50.776132 5474 leader_election.cc:336] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853): Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111)
I20250813 07:54:50.780295 5676 raft_consensus.cc:2466] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c16f21611464c3692d20959f52afb44 in term 2.
I20250813 07:54:50.781104 5475 leader_election.cc:304] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c16f21611464c3692d20959f52afb44, 635d0b56b4884c8e922d86644c5d4be5; no voters: c06e2e4cbc5c4ba7abfeab8037b9e23d
I20250813 07:54:50.781864 5781 raft_consensus.cc:2802] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:54:50.783991 5781 raft_consensus.cc:695] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [term 2 LEADER]: Becoming Leader. State: Replica: 1c16f21611464c3692d20959f52afb44, State: Running, Role: LEADER
I20250813 07:54:50.784965 5781 consensus_queue.cc:237] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } }
I20250813 07:54:50.796627 4999 catalog_manager.cc:5582] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 reported cstate change: term changed from 1 to 2, leader changed from c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193) to 1c16f21611464c3692d20959f52afb44 (127.0.106.196). New cstate: current_term: 2 leader_uuid: "1c16f21611464c3692d20959f52afb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c16f21611464c3692d20959f52afb44" member_type: VOTER last_known_addr { host: "127.0.106.196" port: 46881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c06e2e4cbc5c4ba7abfeab8037b9e23d" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 } health_report { overall_health: UNKNOWN } } }
W20250813 07:54:50.833230 5636 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33826: Illegal state: replica 635d0b56b4884c8e922d86644c5d4be5 is not leader of this config: current role FOLLOWER
I20250813 07:54:50.858712 5255 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:50.866604 5521 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:50.882975 5388 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:50.886520 5656 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:50.934026 5676 raft_consensus.cc:1273] T 0edad7874b8d4607b0ef739b9efe0288 P 635d0b56b4884c8e922d86644c5d4be5 [term 2 FOLLOWER]: Refusing update from remote peer 1c16f21611464c3692d20959f52afb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 07:54:50.935880 5781 consensus_queue.cc:1035] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "635d0b56b4884c8e922d86644c5d4be5" member_type: VOTER last_known_addr { host: "127.0.106.197" port: 34913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 07:54:50.945258 5474 consensus_peers.cc:489] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 -> Peer c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853): Couldn't send request to peer c06e2e4cbc5c4ba7abfeab8037b9e23d. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 07:54:50.987092 5809 mvcc.cc:204] Tried to move back new op lower bound from 7188773646026706944 to 7188773645462364160. Current Snapshot: MvccSnapshot[applied={T|T < 7188773646026706944}]
I20250813 07:54:50.992619 5811 mvcc.cc:204] Tried to move back new op lower bound from 7188773646026706944 to 7188773645462364160. Current Snapshot: MvccSnapshot[applied={T|T < 7188773646026706944}]
I20250813 07:54:52.628461 5521 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:52.646327 5656 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:52.655208 5255 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:52.679793 5388 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250813 07:54:53.621104 5474 consensus_peers.cc:489] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 -> Peer c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853): Couldn't send request to peer c06e2e4cbc5c4ba7abfeab8037b9e23d. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250813 07:54:54.343621 5255 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:54.363538 5388 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:54:54.370903 5521 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 07:54:54.395957 5656 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250813 07:54:55.920670 5474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250813 07:54:55.923822 5474 consensus_peers.cc:489] T 0edad7874b8d4607b0ef739b9efe0288 P 1c16f21611464c3692d20959f52afb44 -> Peer c06e2e4cbc5c4ba7abfeab8037b9e23d (127.0.106.193:44853): Couldn't send request to peer c06e2e4cbc5c4ba7abfeab8037b9e23d. Status: Network error: Client connection negotiation failed: client connection to 127.0.106.193:44853: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250813 07:54:56.918655 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5191
I20250813 07:54:56.939939 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5324
I20250813 07:54:56.961899 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5457
I20250813 07:54:56.998251 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5590
I20250813 07:54:57.026036 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 4965
2025-08-13T07:54:57Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20489 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20489 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250813 07:54:57.082518 427 test_util.cc:276] Using random seed: 2031048830
I20250813 07:54:57.086720 427 ts_itest-base.cc:115] Starting cluster with:
I20250813 07:54:57.086901 427 ts_itest-base.cc:116] --------------
I20250813 07:54:57.087044 427 ts_itest-base.cc:117] 1 tablet servers
I20250813 07:54:57.087198 427 ts_itest-base.cc:118] 1 replicas per TS
I20250813 07:54:57.087322 427 ts_itest-base.cc:119] --------------
2025-08-13T07:54:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:54:57Z Disabled control of system clock
I20250813 07:54:57.129848 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46183
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:46183 with env {}
W20250813 07:54:57.440307 5906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:57.440899 5906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:57.441366 5906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:57.472673 5906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:54:57.473033 5906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:57.473299 5906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:54:57.473585 5906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:54:57.509608 5906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:46183
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:46183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:57.510918 5906 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:57.512595 5906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:57.523319 5912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:57.523765 5913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:58.632484 5915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:58.635372 5914 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
W20250813 07:54:58.635983 5906 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.113s user 0.402s sys 0.708s
W20250813 07:54:58.636227 5906 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.113s user 0.402s sys 0.708s
I20250813 07:54:58.636445 5906 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:54:58.637553 5906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:54:58.640115 5906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:54:58.641508 5906 hybrid_clock.cc:648] HybridClock initialized: now 1755071698641437 us; error 57 us; skew 500 ppm
I20250813 07:54:58.642306 5906 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:54:58.649971 5906 webserver.cc:480] Webserver started at http://127.0.106.254:42933/ using document root <none> and password file <none>
I20250813 07:54:58.650913 5906 fs_manager.cc:362] Metadata directory not provided
I20250813 07:54:58.651152 5906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:54:58.651613 5906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:54:58.656052 5906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6f87ece0af4d481eb04060a9b1805219"
format_stamp: "Formatted at 2025-08-13 07:54:58 on dist-test-slave-dr1z"
I20250813 07:54:58.657177 5906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6f87ece0af4d481eb04060a9b1805219"
format_stamp: "Formatted at 2025-08-13 07:54:58 on dist-test-slave-dr1z"
I20250813 07:54:58.665144 5906 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250813 07:54:58.671288 5922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:54:58.672498 5906 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250813 07:54:58.672844 5906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6f87ece0af4d481eb04060a9b1805219"
format_stamp: "Formatted at 2025-08-13 07:54:58 on dist-test-slave-dr1z"
I20250813 07:54:58.673175 5906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:54:58.754734 5906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:54:58.756253 5906 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:54:58.756701 5906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:54:58.824744 5906 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:46183
I20250813 07:54:58.824810 5973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:46183 every 8 connection(s)
I20250813 07:54:58.827554 5906 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 07:54:58.830786 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5906
I20250813 07:54:58.831290 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 07:54:58.833356 5974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:54:58.857168 5974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219: Bootstrap starting.
I20250813 07:54:58.862828 5974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219: Neither blocks nor log segments found. Creating new log.
I20250813 07:54:58.864439 5974 log.cc:826] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219: Log is configured to *not* fsync() on all Append() calls
I20250813 07:54:58.868925 5974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219: No bootstrap required, opened a new log
I20250813 07:54:58.886605 5974 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } }
I20250813 07:54:58.887241 5974 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:54:58.887457 5974 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f87ece0af4d481eb04060a9b1805219, State: Initialized, Role: FOLLOWER
I20250813 07:54:58.888166 5974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } }
I20250813 07:54:58.888769 5974 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:54:58.889070 5974 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:54:58.889403 5974 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:54:58.893416 5974 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } }
I20250813 07:54:58.894152 5974 leader_election.cc:304] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f87ece0af4d481eb04060a9b1805219; no voters:
I20250813 07:54:58.895907 5974 leader_election.cc:290] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:54:58.896728 5979 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:54:58.898619 5979 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [term 1 LEADER]: Becoming Leader. State: Replica: 6f87ece0af4d481eb04060a9b1805219, State: Running, Role: LEADER
I20250813 07:54:58.899369 5979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } }
I20250813 07:54:58.900206 5974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:54:58.905968 5981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f87ece0af4d481eb04060a9b1805219. Latest consensus state: current_term: 1 leader_uuid: "6f87ece0af4d481eb04060a9b1805219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } } }
I20250813 07:54:58.906670 5981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:58.906744 5980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f87ece0af4d481eb04060a9b1805219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87ece0af4d481eb04060a9b1805219" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 46183 } } }
I20250813 07:54:58.910737 5980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219 [sys.catalog]: This master's current role is: LEADER
I20250813 07:54:58.913884 5986 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:54:58.924789 5986 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:54:58.939852 5986 catalog_manager.cc:1349] Generated new cluster ID: c47044b6058f42e985538d621e35d55f
I20250813 07:54:58.940150 5986 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:54:58.952593 5986 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:54:58.954128 5986 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:54:58.966120 5986 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6f87ece0af4d481eb04060a9b1805219: Generated new TSK 0
I20250813 07:54:58.966984 5986 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:54:58.979984 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46183
--builtin_ntp_servers=127.0.106.212:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 07:54:59.313027 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:54:59.313583 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:54:59.314072 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:54:59.345947 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:54:59.346794 5998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:54:59.381992 5998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:46183
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:54:59.383258 5998 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:54:59.384838 5998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:54:59.397593 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:59.400487 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:54:59.402552 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:00.871982 6006 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1469 milliseconds
I20250813 07:55:00.872107 5998 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:00.873318 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:00.875870 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:00.877283 5998 hybrid_clock.cc:648] HybridClock initialized: now 1755071700877244 us; error 53 us; skew 500 ppm
I20250813 07:55:00.878090 5998 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:00.884629 5998 webserver.cc:480] Webserver started at http://127.0.106.193:39905/ using document root <none> and password file <none>
I20250813 07:55:00.885600 5998 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:00.885812 5998 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:00.886267 5998 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:00.890576 5998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "335a79f6d5fd45708da89dd254682ec9"
format_stamp: "Formatted at 2025-08-13 07:55:00 on dist-test-slave-dr1z"
I20250813 07:55:00.891688 5998 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "335a79f6d5fd45708da89dd254682ec9"
format_stamp: "Formatted at 2025-08-13 07:55:00 on dist-test-slave-dr1z"
I20250813 07:55:00.898757 5998 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250813 07:55:00.904567 6014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:00.905582 5998 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:55:00.905898 5998 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "335a79f6d5fd45708da89dd254682ec9"
format_stamp: "Formatted at 2025-08-13 07:55:00 on dist-test-slave-dr1z"
I20250813 07:55:00.906227 5998 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:00.959841 5998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:00.961337 5998 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:00.961814 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:00.964501 5998 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:00.968760 5998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:55:00.968974 5998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:00.969182 5998 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:55:00.969316 5998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:01.096889 5998 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:38709
I20250813 07:55:01.097009 6126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:38709 every 8 connection(s)
I20250813 07:55:01.099642 5998 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 07:55:01.108500 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 5998
I20250813 07:55:01.108907 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755071556474372-427-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 07:55:01.120779 6127 heartbeater.cc:344] Connected to a master server at 127.0.106.254:46183
I20250813 07:55:01.121213 6127 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:01.122274 6127 heartbeater.cc:507] Master 127.0.106.254:46183 requested a full tablet report, sending...
I20250813 07:55:01.124745 5939 ts_manager.cc:194] Registered new tserver with Master: 335a79f6d5fd45708da89dd254682ec9 (127.0.106.193:38709)
I20250813 07:55:01.126870 5939 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:53815
I20250813 07:55:01.128651 427 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250813 07:55:01.159711 5938 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39348:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 07:55:01.217097 6062 tablet_service.cc:1468] Processing CreateTablet for tablet 8dc92a49c3154ca3ad25243a7b38d64e (DEFAULT_TABLE table=TestTable [id=7dcddf2a8bef425c94b6fb243ff5722e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:01.218770 6062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dc92a49c3154ca3ad25243a7b38d64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:01.238843 6142 tablet_bootstrap.cc:492] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: Bootstrap starting.
I20250813 07:55:01.244436 6142 tablet_bootstrap.cc:654] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:01.246227 6142 log.cc:826] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:01.251107 6142 tablet_bootstrap.cc:492] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: No bootstrap required, opened a new log
I20250813 07:55:01.251521 6142 ts_tablet_manager.cc:1397] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250813 07:55:01.268741 6142 raft_consensus.cc:357] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335a79f6d5fd45708da89dd254682ec9" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38709 } }
I20250813 07:55:01.269315 6142 raft_consensus.cc:383] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:01.269589 6142 raft_consensus.cc:738] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335a79f6d5fd45708da89dd254682ec9, State: Initialized, Role: FOLLOWER
I20250813 07:55:01.270291 6142 consensus_queue.cc:260] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335a79f6d5fd45708da89dd254682ec9" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38709 } }
I20250813 07:55:01.270819 6142 raft_consensus.cc:397] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:55:01.271095 6142 raft_consensus.cc:491] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:55:01.271406 6142 raft_consensus.cc:3058] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:01.275895 6142 raft_consensus.cc:513] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335a79f6d5fd45708da89dd254682ec9" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38709 } }
I20250813 07:55:01.276830 6142 leader_election.cc:304] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 335a79f6d5fd45708da89dd254682ec9; no voters:
I20250813 07:55:01.278549 6142 leader_election.cc:290] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:55:01.278954 6144 raft_consensus.cc:2802] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:55:01.281597 6144 raft_consensus.cc:695] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [term 1 LEADER]: Becoming Leader. State: Replica: 335a79f6d5fd45708da89dd254682ec9, State: Running, Role: LEADER
I20250813 07:55:01.282193 6127 heartbeater.cc:499] Master 127.0.106.254:46183 was elected leader, sending a full tablet report...
I20250813 07:55:01.282482 6144 consensus_queue.cc:237] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335a79f6d5fd45708da89dd254682ec9" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38709 } }
I20250813 07:55:01.283226 6142 ts_tablet_manager.cc:1428] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9: Time spent starting tablet: real 0.031s user 0.020s sys 0.012s
I20250813 07:55:01.295181 5938 catalog_manager.cc:5582] T 8dc92a49c3154ca3ad25243a7b38d64e P 335a79f6d5fd45708da89dd254682ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 335a79f6d5fd45708da89dd254682ec9 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "335a79f6d5fd45708da89dd254682ec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "335a79f6d5fd45708da89dd254682ec9" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 38709 } health_report { overall_health: HEALTHY } } }
I20250813 07:55:01.323004 427 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250813 07:55:01.326023 427 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 335a79f6d5fd45708da89dd254682ec9 to finish bootstrapping
I20250813 07:55:03.952325 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5998
I20250813 07:55:03.975147 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 5906
2025-08-13T07:55:03Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6947 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6947 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250813 07:55:04.030886 427 test_util.cc:276] Using random seed: 2037997192
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (13 ms)
[----------] 1 test from ListTableCliParamTest (13 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-13T07:55:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T07:55:04Z Disabled control of system clock
I20250813 07:55:04.083961 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:36111
--webserver_interface=127.0.106.254
--webserver_port=0
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:36111 with env {}
W20250813 07:55:04.397497 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:04.398165 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:04.398654 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:04.432015 6170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:55:04.432348 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:04.432559 6170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:55:04.432765 6170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:55:04.470341 6170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:36111
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:04.471697 6170 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:04.473585 6170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:04.485692 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:05.888482 6175 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 6170
W20250813 07:55:06.058281 6170 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.573s user 0.625s sys 0.946s
W20250813 07:55:06.058640 6170 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.573s user 0.625s sys 0.946s
W20250813 07:55:04.486845 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:06.060556 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:06.063019 6178 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1572 milliseconds
I20250813 07:55:06.063092 6170 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:06.064247 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:06.066691 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:06.068027 6170 hybrid_clock.cc:648] HybridClock initialized: now 1755071706067968 us; error 64 us; skew 500 ppm
I20250813 07:55:06.068810 6170 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:06.074779 6170 webserver.cc:480] Webserver started at http://127.0.106.254:37563/ using document root <none> and password file <none>
I20250813 07:55:06.075711 6170 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:06.075925 6170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:06.076421 6170 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:06.080790 6170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/instance:
uuid: "85bfbb1b47fc4931bd67da430b306929"
format_stamp: "Formatted at 2025-08-13 07:55:06 on dist-test-slave-dr1z"
I20250813 07:55:06.081960 6170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal/instance:
uuid: "85bfbb1b47fc4931bd67da430b306929"
format_stamp: "Formatted at 2025-08-13 07:55:06 on dist-test-slave-dr1z"
I20250813 07:55:06.089175 6170 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250813 07:55:06.094947 6186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:06.096099 6170 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250813 07:55:06.096544 6170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
uuid: "85bfbb1b47fc4931bd67da430b306929"
format_stamp: "Formatted at 2025-08-13 07:55:06 on dist-test-slave-dr1z"
I20250813 07:55:06.096936 6170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:06.163969 6170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:06.165535 6170 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:06.165978 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:06.233884 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:36111
I20250813 07:55:06.233958 6237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:36111 every 8 connection(s)
I20250813 07:55:06.236603 6170 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
I20250813 07:55:06.240512 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6170
I20250813 07:55:06.241030 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal/instance
I20250813 07:55:06.242615 6238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:06.267472 6238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929: Bootstrap starting.
I20250813 07:55:06.272851 6238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:06.274668 6238 log.cc:826] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:06.279033 6238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929: No bootstrap required, opened a new log
I20250813 07:55:06.296836 6238 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:06.297538 6238 raft_consensus.cc:383] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:06.297783 6238 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bfbb1b47fc4931bd67da430b306929, State: Initialized, Role: FOLLOWER
I20250813 07:55:06.298437 6238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:06.298940 6238 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:55:06.299201 6238 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:55:06.299508 6238 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:06.303516 6238 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:06.304248 6238 leader_election.cc:304] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bfbb1b47fc4931bd67da430b306929; no voters:
I20250813 07:55:06.306154 6238 leader_election.cc:290] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:55:06.306865 6243 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:55:06.308861 6243 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [term 1 LEADER]: Becoming Leader. State: Replica: 85bfbb1b47fc4931bd67da430b306929, State: Running, Role: LEADER
I20250813 07:55:06.309649 6243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:06.310568 6238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:55:06.319947 6245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bfbb1b47fc4931bd67da430b306929. Latest consensus state: current_term: 1 leader_uuid: "85bfbb1b47fc4931bd67da430b306929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } } }
I20250813 07:55:06.320739 6245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:06.321131 6244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bfbb1b47fc4931bd67da430b306929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bfbb1b47fc4931bd67da430b306929" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } } }
I20250813 07:55:06.321856 6244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:06.323833 6251 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:55:06.334146 6251 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:55:06.351486 6251 catalog_manager.cc:1349] Generated new cluster ID: a2e330aeba9946e8bd745f6bad241556
I20250813 07:55:06.351781 6251 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:55:06.370904 6251 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:55:06.372424 6251 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:55:06.384006 6251 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 85bfbb1b47fc4931bd67da430b306929: Generated new TSK 0
I20250813 07:55:06.384891 6251 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 07:55:06.407214 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:0
--local_ip_for_outbound_sockets=127.0.106.193
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250813 07:55:06.757256 6262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:06.757828 6262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:06.758319 6262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:06.791646 6262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:06.792558 6262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:55:06.833946 6262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:06.835486 6262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:06.837337 6262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:06.853047 6268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:08.254837 6267 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6262
W20250813 07:55:08.327329 6262 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.471s user 0.557s sys 0.910s
W20250813 07:55:08.327685 6262 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.472s user 0.557s sys 0.910s
W20250813 07:55:06.856947 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:08.329039 6271 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1471 milliseconds
W20250813 07:55:08.331265 6272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:08.331261 6262 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:08.340034 6262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:08.345016 6262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:08.349013 6262 hybrid_clock.cc:648] HybridClock initialized: now 1755071708348905 us; error 152 us; skew 500 ppm
I20250813 07:55:08.350049 6262 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:08.358024 6262 webserver.cc:480] Webserver started at http://127.0.106.193:36905/ using document root <none> and password file <none>
I20250813 07:55:08.359030 6262 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:08.359243 6262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:08.359725 6262 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:08.364878 6262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/instance:
uuid: "d02ee4f03a91457d9f197c6da4ef75b0"
format_stamp: "Formatted at 2025-08-13 07:55:08 on dist-test-slave-dr1z"
I20250813 07:55:08.366323 6262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal/instance:
uuid: "d02ee4f03a91457d9f197c6da4ef75b0"
format_stamp: "Formatted at 2025-08-13 07:55:08 on dist-test-slave-dr1z"
I20250813 07:55:08.375569 6262 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250813 07:55:08.382169 6279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:08.383358 6262 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250813 07:55:08.383692 6262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
uuid: "d02ee4f03a91457d9f197c6da4ef75b0"
format_stamp: "Formatted at 2025-08-13 07:55:08 on dist-test-slave-dr1z"
I20250813 07:55:08.384029 6262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:08.453576 6262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:08.455072 6262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:08.455489 6262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:08.458353 6262 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:08.462577 6262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:55:08.462785 6262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:08.463044 6262 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:55:08.463193 6262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:08.620760 6262 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:36963
I20250813 07:55:08.620886 6391 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:36963 every 8 connection(s)
I20250813 07:55:08.623504 6262 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
I20250813 07:55:08.629119 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6262
I20250813 07:55:08.629971 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal/instance
I20250813 07:55:08.639600 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:0
--local_ip_for_outbound_sockets=127.0.106.194
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 07:55:08.657909 6392 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:08.658363 6392 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:08.659440 6392 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:08.662173 6203 ts_manager.cc:194] Registered new tserver with Master: d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193:36963)
I20250813 07:55:08.664191 6203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:52775
W20250813 07:55:08.971546 6396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:08.972075 6396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:08.972571 6396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:09.004912 6396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:09.005851 6396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:55:09.041846 6396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:09.043129 6396 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:09.044693 6396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:09.056228 6402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:09.667819 6392 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
W20250813 07:55:10.459319 6401 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 6396
W20250813 07:55:10.571771 6396 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.515s user 0.646s sys 0.868s
W20250813 07:55:09.057412 6403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:10.572160 6396 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.515s user 0.647s sys 0.868s
W20250813 07:55:10.574379 6405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:10.576735 6404 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1516 milliseconds
I20250813 07:55:10.576804 6396 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:10.578014 6396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:10.580071 6396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:10.581435 6396 hybrid_clock.cc:648] HybridClock initialized: now 1755071710581399 us; error 48 us; skew 500 ppm
I20250813 07:55:10.582249 6396 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:10.588142 6396 webserver.cc:480] Webserver started at http://127.0.106.194:44517/ using document root <none> and password file <none>
I20250813 07:55:10.589094 6396 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:10.589318 6396 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:10.589773 6396 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:10.594133 6396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/instance:
uuid: "491f917a421b40c7816aa138436918e1"
format_stamp: "Formatted at 2025-08-13 07:55:10 on dist-test-slave-dr1z"
I20250813 07:55:10.595221 6396 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal/instance:
uuid: "491f917a421b40c7816aa138436918e1"
format_stamp: "Formatted at 2025-08-13 07:55:10 on dist-test-slave-dr1z"
I20250813 07:55:10.602217 6396 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250813 07:55:10.608016 6412 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:10.609083 6396 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 07:55:10.609401 6396 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
uuid: "491f917a421b40c7816aa138436918e1"
format_stamp: "Formatted at 2025-08-13 07:55:10 on dist-test-slave-dr1z"
I20250813 07:55:10.609758 6396 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:10.658483 6396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:10.659899 6396 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:10.660315 6396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:10.662750 6396 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:10.666703 6396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:55:10.666916 6396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:10.667151 6396 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:55:10.667311 6396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:10.795066 6396 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:34467
I20250813 07:55:10.795184 6524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:34467 every 8 connection(s)
I20250813 07:55:10.797567 6396 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
I20250813 07:55:10.802098 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6396
I20250813 07:55:10.802588 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal/instance
I20250813 07:55:10.809209 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:0
--local_ip_for_outbound_sockets=127.0.106.195
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 07:55:10.819249 6525 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:10.819738 6525 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:10.821050 6525 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:10.823359 6203 ts_manager.cc:194] Registered new tserver with Master: 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467)
I20250813 07:55:10.824632 6203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:55391
W20250813 07:55:11.108625 6529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:11.109138 6529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:11.109680 6529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:11.141036 6529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:11.142287 6529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:55:11.177877 6529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=0
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:11.179153 6529 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:11.180734 6529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:11.192767 6535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:11.828419 6525 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
W20250813 07:55:11.193739 6536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:11.197038 6538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:12.274881 6537 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250813 07:55:12.274935 6529 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:12.278753 6529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:12.281389 6529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:12.282860 6529 hybrid_clock.cc:648] HybridClock initialized: now 1755071712282793 us; error 82 us; skew 500 ppm
I20250813 07:55:12.283681 6529 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:12.290258 6529 webserver.cc:480] Webserver started at http://127.0.106.195:37733/ using document root <none> and password file <none>
I20250813 07:55:12.291198 6529 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:12.291397 6529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:12.291867 6529 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:12.296296 6529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/instance:
uuid: "7877484fe82e4183b76bd7f1830f7932"
format_stamp: "Formatted at 2025-08-13 07:55:12 on dist-test-slave-dr1z"
I20250813 07:55:12.297410 6529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal/instance:
uuid: "7877484fe82e4183b76bd7f1830f7932"
format_stamp: "Formatted at 2025-08-13 07:55:12 on dist-test-slave-dr1z"
I20250813 07:55:12.304575 6529 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250813 07:55:12.310407 6545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:12.311545 6529 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:55:12.311857 6529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
uuid: "7877484fe82e4183b76bd7f1830f7932"
format_stamp: "Formatted at 2025-08-13 07:55:12 on dist-test-slave-dr1z"
I20250813 07:55:12.312183 6529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:12.361610 6529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:12.363009 6529 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:12.363639 6529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:12.366374 6529 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:12.370469 6529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 07:55:12.370683 6529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:12.370925 6529 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 07:55:12.371091 6529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:12.499873 6529 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:39465
I20250813 07:55:12.499972 6657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:39465 every 8 connection(s)
I20250813 07:55:12.502389 6529 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
I20250813 07:55:12.502873 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6529
I20250813 07:55:12.503276 427 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal/instance
I20250813 07:55:12.527431 6658 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:12.527861 6658 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:12.528860 6658 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:12.530974 6203 ts_manager.cc:194] Registered new tserver with Master: 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:12.532234 6203 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:41093
I20250813 07:55:12.538324 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:55:12.568392 427 test_util.cc:276] Using random seed: 2046534704
I20250813 07:55:12.609272 6203 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56446:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250813 07:55:12.611832 6203 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:55:12.661495 6327 tablet_service.cc:1468] Processing CreateTablet for tablet 1b03be001ebc40fdaff20ae647fa957d (DEFAULT_TABLE table=pre_rebuild [id=e560f8a8f4c848d497f081f1afdb313e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:12.663592 6327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b03be001ebc40fdaff20ae647fa957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:12.661602 6460 tablet_service.cc:1468] Processing CreateTablet for tablet 1b03be001ebc40fdaff20ae647fa957d (DEFAULT_TABLE table=pre_rebuild [id=e560f8a8f4c848d497f081f1afdb313e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:12.665166 6460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b03be001ebc40fdaff20ae647fa957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:12.669787 6593 tablet_service.cc:1468] Processing CreateTablet for tablet 1b03be001ebc40fdaff20ae647fa957d (DEFAULT_TABLE table=pre_rebuild [id=e560f8a8f4c848d497f081f1afdb313e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:12.671710 6593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b03be001ebc40fdaff20ae647fa957d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:12.689231 6682 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Bootstrap starting.
I20250813 07:55:12.695578 6683 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Bootstrap starting.
I20250813 07:55:12.696792 6682 tablet_bootstrap.cc:654] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:12.697556 6684 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Bootstrap starting.
I20250813 07:55:12.699350 6682 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:12.704042 6683 tablet_bootstrap.cc:654] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:12.704136 6684 tablet_bootstrap.cc:654] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:12.704914 6682 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: No bootstrap required, opened a new log
I20250813 07:55:12.705453 6682 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.010s
I20250813 07:55:12.705868 6683 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:12.706336 6684 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:12.710994 6684 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: No bootstrap required, opened a new log
I20250813 07:55:12.711221 6683 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: No bootstrap required, opened a new log
I20250813 07:55:12.711426 6684 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20250813 07:55:12.711719 6683 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20250813 07:55:12.729868 6684 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.730572 6684 raft_consensus.cc:383] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:12.730818 6684 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877484fe82e4183b76bd7f1830f7932, State: Initialized, Role: FOLLOWER
I20250813 07:55:12.731613 6684 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.732457 6682 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.733376 6682 raft_consensus.cc:383] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:12.733701 6682 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Initialized, Role: FOLLOWER
I20250813 07:55:12.734627 6682 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.738942 6684 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Time spent starting tablet: real 0.027s user 0.012s sys 0.015s
I20250813 07:55:12.739444 6682 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Time spent starting tablet: real 0.034s user 0.028s sys 0.004s
I20250813 07:55:12.738883 6683 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.739869 6683 raft_consensus.cc:383] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:12.739933 6658 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
I20250813 07:55:12.740204 6683 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491f917a421b40c7816aa138436918e1, State: Initialized, Role: FOLLOWER
I20250813 07:55:12.741082 6683 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.745163 6683 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Time spent starting tablet: real 0.033s user 0.030s sys 0.003s
W20250813 07:55:12.756902 6659 tablet.cc:2378] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:55:12.803534 6526 tablet.cc:2378] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 07:55:12.882387 6393 tablet.cc:2378] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:55:12.979424 6689 raft_consensus.cc:491] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:12.979977 6689 raft_consensus.cc:513] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:12.982465 6689 leader_election.cc:290] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:12.993966 6480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1" is_pre_election: true
I20250813 07:55:12.994684 6480 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 0.
I20250813 07:55:12.995636 6613 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932" is_pre_election: true
I20250813 07:55:12.995838 6282 leader_election.cc:304] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:12.996258 6613 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 0.
I20250813 07:55:12.996594 6689 raft_consensus.cc:2802] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:55:12.996861 6689 raft_consensus.cc:491] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:55:12.997159 6689 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:13.001448 6689 raft_consensus.cc:513] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:13.002857 6689 leader_election.cc:290] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 election: Requested vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:13.003770 6480 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1"
I20250813 07:55:13.004065 6613 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932"
I20250813 07:55:13.004316 6480 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:13.004585 6613 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:13.011070 6613 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 1.
I20250813 07:55:13.011068 6480 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 1.
I20250813 07:55:13.012102 6282 leader_election.cc:304] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7877484fe82e4183b76bd7f1830f7932, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:13.012791 6689 raft_consensus.cc:2802] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:55:13.014312 6689 raft_consensus.cc:695] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 LEADER]: Becoming Leader. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Running, Role: LEADER
I20250813 07:55:13.015070 6689 consensus_queue.cc:237] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:13.025099 6202 catalog_manager.cc:5582] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20250813 07:55:13.276273 6613 raft_consensus.cc:1273] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 07:55:13.276433 6480 raft_consensus.cc:1273] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 07:55:13.278074 6689 consensus_queue.cc:1035] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:55:13.278764 6693 consensus_queue.cc:1035] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:55:13.313890 6702 mvcc.cc:204] Tried to move back new op lower bound from 7188773737562828800 to 7188773736516878336. Current Snapshot: MvccSnapshot[applied={T|T < 7188773737562828800}]
I20250813 07:55:13.319517 6706 mvcc.cc:204] Tried to move back new op lower bound from 7188773737562828800 to 7188773736516878336. Current Snapshot: MvccSnapshot[applied={T|T < 7188773737562828800}]
W20250813 07:55:15.896636 6654 debug-util.cc:398] Leaking SignalData structure 0x7b08000837c0 after lost signal to thread 6530
W20250813 07:55:15.897848 6654 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccc60 after lost signal to thread 6657
W20250813 07:55:17.905721 6698 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.000s sys 0.000s
I20250813 07:55:19.092149 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6170
W20250813 07:55:19.386018 6525 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:36111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:36111: connect: Connection refused (error 111)
W20250813 07:55:19.394797 6392 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:36111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:36111: connect: Connection refused (error 111)
W20250813 07:55:19.411170 6658 heartbeater.cc:646] Failed to heartbeat to 127.0.106.254:36111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.106.254:36111: connect: Connection refused (error 111)
W20250813 07:55:19.451205 6742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:19.452111 6742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:19.485285 6742 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 07:55:20.842886 6742 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.318s user 0.552s sys 0.761s
W20250813 07:55:20.843304 6742 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.319s user 0.553s sys 0.762s
I20250813 07:55:20.958737 6742 minidump.cc:252] Setting minidump size limit to 20M
I20250813 07:55:20.961143 6742 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:20.962765 6742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:20.976189 6774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:20.977883 6775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:21.120543 6742 server_base.cc:1047] running on GCE node
W20250813 07:55:21.120586 6777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:21.121917 6742 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250813 07:55:21.122478 6742 hybrid_clock.cc:648] HybridClock initialized: now 1755071721122448 us; error 125016 us; skew 500 ppm
I20250813 07:55:21.123325 6742 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:21.128568 6742 webserver.cc:480] Webserver started at http://0.0.0.0:39849/ using document root <none> and password file <none>
I20250813 07:55:21.129689 6742 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:21.129971 6742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:21.130527 6742 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 07:55:21.135767 6742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/instance:
uuid: "27737bcae690471797cea55dfb8fe534"
format_stamp: "Formatted at 2025-08-13 07:55:21 on dist-test-slave-dr1z"
I20250813 07:55:21.136948 6742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal/instance:
uuid: "27737bcae690471797cea55dfb8fe534"
format_stamp: "Formatted at 2025-08-13 07:55:21 on dist-test-slave-dr1z"
I20250813 07:55:21.143056 6742 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250813 07:55:21.148226 6783 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:21.149128 6742 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250813 07:55:21.149430 6742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
uuid: "27737bcae690471797cea55dfb8fe534"
format_stamp: "Formatted at 2025-08-13 07:55:21 on dist-test-slave-dr1z"
I20250813 07:55:21.149775 6742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:21.172312 6742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:21.173698 6742 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:21.174120 6742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:21.178550 6742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:21.192314 6742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Bootstrap starting.
I20250813 07:55:21.197393 6742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:21.199116 6742 log.cc:826] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:21.203066 6742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: No bootstrap required, opened a new log
I20250813 07:55:21.219799 6742 raft_consensus.cc:357] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER }
I20250813 07:55:21.220278 6742 raft_consensus.cc:383] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:21.220516 6742 raft_consensus.cc:738] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27737bcae690471797cea55dfb8fe534, State: Initialized, Role: FOLLOWER
I20250813 07:55:21.221133 6742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER }
I20250813 07:55:21.221621 6742 raft_consensus.cc:397] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:55:21.221851 6742 raft_consensus.cc:491] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:55:21.222132 6742 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:21.226060 6742 raft_consensus.cc:513] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER }
I20250813 07:55:21.226711 6742 leader_election.cc:304] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27737bcae690471797cea55dfb8fe534; no voters:
I20250813 07:55:21.228314 6742 leader_election.cc:290] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 07:55:21.228554 6790 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:55:21.230638 6790 raft_consensus.cc:695] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 LEADER]: Becoming Leader. State: Replica: 27737bcae690471797cea55dfb8fe534, State: Running, Role: LEADER
I20250813 07:55:21.231530 6790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER }
I20250813 07:55:21.237802 6792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27737bcae690471797cea55dfb8fe534. Latest consensus state: current_term: 1 leader_uuid: "27737bcae690471797cea55dfb8fe534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER } }
I20250813 07:55:21.238035 6791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27737bcae690471797cea55dfb8fe534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER } }
I20250813 07:55:21.238445 6792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:21.238526 6791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:21.252249 6742 tablet_replica.cc:331] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: stopping tablet replica
I20250813 07:55:21.252754 6742 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 LEADER]: Raft consensus shutting down.
I20250813 07:55:21.253129 6742 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250813 07:55:21.255287 6742 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250813 07:55:21.255718 6742 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250813 07:55:21.280990 6742 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250813 07:55:22.309382 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6262
I20250813 07:55:22.345906 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6396
I20250813 07:55:22.372501 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6529
I20250813 07:55:22.401818 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:36111
--webserver_interface=127.0.106.254
--webserver_port=37563
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.106.254:36111 with env {}
W20250813 07:55:22.699371 6799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:22.699956 6799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:22.700402 6799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:22.730994 6799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 07:55:22.731302 6799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:22.731582 6799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 07:55:22.731835 6799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 07:55:22.767565 6799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.106.254:36111
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.106.254:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.106.254
--webserver_port=37563
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:22.768905 6799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:22.770500 6799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:22.780762 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:22.782123 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:23.899582 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:23.902208 6807 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1115 milliseconds
I20250813 07:55:23.902355 6799 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:23.903692 6799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:23.906217 6799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:23.907617 6799 hybrid_clock.cc:648] HybridClock initialized: now 1755071723907541 us; error 46 us; skew 500 ppm
I20250813 07:55:23.908399 6799 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:23.915066 6799 webserver.cc:480] Webserver started at http://127.0.106.254:37563/ using document root <none> and password file <none>
I20250813 07:55:23.916003 6799 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:23.916231 6799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:23.924043 6799 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250813 07:55:23.928577 6815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:23.929651 6799 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250813 07:55:23.929996 6799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
uuid: "27737bcae690471797cea55dfb8fe534"
format_stamp: "Formatted at 2025-08-13 07:55:21 on dist-test-slave-dr1z"
I20250813 07:55:23.931952 6799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:23.990294 6799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:23.991931 6799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:23.992448 6799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:24.063138 6799 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.254:36111
I20250813 07:55:24.063186 6866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.254:36111 every 8 connection(s)
I20250813 07:55:24.066080 6799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb
I20250813 07:55:24.075140 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6799
I20250813 07:55:24.076237 6867 sys_catalog.cc:263] Verifying existing consensus state
I20250813 07:55:24.076664 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.193:36963
--local_ip_for_outbound_sockets=127.0.106.193
--tserver_master_addrs=127.0.106.254:36111
--webserver_port=36905
--webserver_interface=127.0.106.193
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 07:55:24.088303 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Bootstrap starting.
I20250813 07:55:24.098222 6867 log.cc:826] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:24.110208 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:55:24.110970 6867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Bootstrap complete.
I20250813 07:55:24.131105 6867 raft_consensus.cc:357] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:24.131758 6867 raft_consensus.cc:738] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27737bcae690471797cea55dfb8fe534, State: Initialized, Role: FOLLOWER
I20250813 07:55:24.132509 6867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:24.133016 6867 raft_consensus.cc:397] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 07:55:24.133275 6867 raft_consensus.cc:491] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 07:55:24.133601 6867 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:55:24.138055 6867 raft_consensus.cc:513] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:24.138751 6867 leader_election.cc:304] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27737bcae690471797cea55dfb8fe534; no voters:
I20250813 07:55:24.140703 6867 leader_election.cc:290] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 07:55:24.141252 6871 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 07:55:24.144357 6871 raft_consensus.cc:695] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [term 2 LEADER]: Becoming Leader. State: Replica: 27737bcae690471797cea55dfb8fe534, State: Running, Role: LEADER
I20250813 07:55:24.145236 6871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } }
I20250813 07:55:24.145937 6867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 07:55:24.155516 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "27737bcae690471797cea55dfb8fe534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } } }
I20250813 07:55:24.156268 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:24.158350 6873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27737bcae690471797cea55dfb8fe534. Latest consensus state: current_term: 2 leader_uuid: "27737bcae690471797cea55dfb8fe534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27737bcae690471797cea55dfb8fe534" member_type: VOTER last_known_addr { host: "127.0.106.254" port: 36111 } } }
I20250813 07:55:24.159015 6873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534 [sys.catalog]: This master's current role is: LEADER
I20250813 07:55:24.167277 6877 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 07:55:24.178433 6877 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=b2748bdd06414a8394a50b8b49ed7370]
I20250813 07:55:24.186759 6877 tablet_loader.cc:96] loaded metadata for tablet 1b03be001ebc40fdaff20ae647fa957d (table pre_rebuild [id=b2748bdd06414a8394a50b8b49ed7370])
I20250813 07:55:24.188446 6877 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 07:55:24.219733 6877 catalog_manager.cc:1349] Generated new cluster ID: 3945f69960384952ba4ab1e716ee9266
I20250813 07:55:24.220026 6877 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 07:55:24.241308 6877 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 07:55:24.243438 6877 catalog_manager.cc:1506] Loading token signing keys...
I20250813 07:55:24.261384 6877 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Generated new TSK 0
I20250813 07:55:24.262585 6877 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 07:55:24.419617 6869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:24.420115 6869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:24.420603 6869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:24.453051 6869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:24.453989 6869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.193
I20250813 07:55:24.488896 6869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.193:36963
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.106.193
--webserver_port=36905
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.193
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:24.490209 6869 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:24.491770 6869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:24.503901 6896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:24.507946 6899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:24.506376 6897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:25.612244 6869 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 07:55:25.612214 6898 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250813 07:55:25.616597 6869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:25.619277 6869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:25.620690 6869 hybrid_clock.cc:648] HybridClock initialized: now 1755071725620629 us; error 78 us; skew 500 ppm
I20250813 07:55:25.621529 6869 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:25.628018 6869 webserver.cc:480] Webserver started at http://127.0.106.193:36905/ using document root <none> and password file <none>
I20250813 07:55:25.628985 6869 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:25.629196 6869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:25.637290 6869 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250813 07:55:25.642280 6906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:25.643393 6869 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250813 07:55:25.643711 6869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
uuid: "d02ee4f03a91457d9f197c6da4ef75b0"
format_stamp: "Formatted at 2025-08-13 07:55:08 on dist-test-slave-dr1z"
I20250813 07:55:25.645656 6869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:25.704728 6869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:25.706719 6869 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:25.707480 6869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:25.710242 6869 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:25.716437 6913 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 07:55:25.723783 6869 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:55:25.723994 6869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250813 07:55:25.724290 6869 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:55:25.728780 6869 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:55:25.729038 6869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250813 07:55:25.729408 6913 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Bootstrap starting.
I20250813 07:55:25.900978 6869 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.193:36963
I20250813 07:55:25.901199 7019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.193:36963 every 8 connection(s)
I20250813 07:55:25.904682 6869 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb
I20250813 07:55:25.909346 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 6869
I20250813 07:55:25.911185 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.194:34467
--local_ip_for_outbound_sockets=127.0.106.194
--tserver_master_addrs=127.0.106.254:36111
--webserver_port=44517
--webserver_interface=127.0.106.194
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 07:55:25.960341 7020 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:25.960834 7020 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:25.962019 7020 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:25.966813 6832 ts_manager.cc:194] Registered new tserver with Master: d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193:36963)
I20250813 07:55:25.974563 6832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.193:45381
I20250813 07:55:26.035612 6913 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Log is configured to *not* fsync() on all Append() calls
W20250813 07:55:26.245820 7024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:26.246331 7024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:26.246817 7024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:26.278870 7024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:26.279747 7024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.194
I20250813 07:55:26.314577 7024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.194:34467
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.106.194
--webserver_port=44517
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.194
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:26.315847 7024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:26.317520 7024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:26.329855 7031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:26.979681 7020 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
W20250813 07:55:26.332284 7032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:26.333724 7034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:27.409932 7033 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250813 07:55:27.410181 7024 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:27.414309 7024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:27.416975 7024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:27.418538 7024 hybrid_clock.cc:648] HybridClock initialized: now 1755071727418490 us; error 53 us; skew 500 ppm
I20250813 07:55:27.419595 7024 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:27.426771 7024 webserver.cc:480] Webserver started at http://127.0.106.194:44517/ using document root <none> and password file <none>
I20250813 07:55:27.427995 7024 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:27.428292 7024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:27.438601 7024 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250813 07:55:27.444271 7042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:27.445415 7024 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 07:55:27.445820 7024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
uuid: "491f917a421b40c7816aa138436918e1"
format_stamp: "Formatted at 2025-08-13 07:55:10 on dist-test-slave-dr1z"
I20250813 07:55:27.448529 7024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:27.501017 7024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:27.502465 7024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:27.502880 7024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:27.505694 7024 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:27.511672 7049 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 07:55:27.519258 7024 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:55:27.519475 7024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250813 07:55:27.519742 7024 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:55:27.524240 7024 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:55:27.524488 7024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250813 07:55:27.524830 7049 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Bootstrap starting.
I20250813 07:55:27.694154 7024 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.194:34467
I20250813 07:55:27.694362 7155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.194:34467 every 8 connection(s)
I20250813 07:55:27.696872 7024 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb
I20250813 07:55:27.704537 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 7024
I20250813 07:55:27.706773 427 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
/tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.106.195:39465
--local_ip_for_outbound_sockets=127.0.106.195
--tserver_master_addrs=127.0.106.254:36111
--webserver_port=37733
--webserver_interface=127.0.106.195
--builtin_ntp_servers=127.0.106.212:38827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 07:55:27.775439 7156 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:27.775962 7156 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:27.777282 7156 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:27.781836 6832 ts_manager.cc:194] Registered new tserver with Master: 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467)
I20250813 07:55:27.784725 6832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.194:55369
I20250813 07:55:27.828586 7049 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Log is configured to *not* fsync() on all Append() calls
W20250813 07:55:28.084841 7160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 07:55:28.085547 7160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 07:55:28.086346 7160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 07:55:28.121148 7160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 07:55:28.122030 7160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.106.195
I20250813 07:55:28.157094 7160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.106.212:38827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.106.195:39465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.106.195
--webserver_port=37733
--tserver_master_addrs=127.0.106.254:36111
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.106.195
--log_dir=/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3a2f29a92790ee949f8efdb6f299dc52fb672fe6
build type FASTDEBUG
built by None at 13 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7545
TSAN enabled
I20250813 07:55:28.158375 7160 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 07:55:28.159895 7160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 07:55:28.174031 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 07:55:28.218784 6913 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:55:28.219947 6913 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Bootstrap complete.
I20250813 07:55:28.221835 6913 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Time spent bootstrapping tablet: real 2.493s user 2.396s sys 0.088s
I20250813 07:55:28.239555 6913 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:28.243057 6913 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Initialized, Role: FOLLOWER
I20250813 07:55:28.244210 6913 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:28.259622 6913 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Time spent starting tablet: real 0.037s user 0.018s sys 0.017s
I20250813 07:55:28.788993 7156 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
W20250813 07:55:28.189216 7170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:28.181948 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 07:55:29.616391 7169 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1437 milliseconds
I20250813 07:55:29.616531 7160 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 07:55:29.618007 7160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 07:55:29.620604 7160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 07:55:29.622072 7160 hybrid_clock.cc:648] HybridClock initialized: now 1755071729622024 us; error 48 us; skew 500 ppm
I20250813 07:55:29.623103 7160 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 07:55:29.630333 7160 webserver.cc:480] Webserver started at http://127.0.106.195:37733/ using document root <none> and password file <none>
I20250813 07:55:29.631569 7160 fs_manager.cc:362] Metadata directory not provided
I20250813 07:55:29.631847 7160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 07:55:29.642174 7160 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s
I20250813 07:55:29.647954 7178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 07:55:29.649147 7160 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20250813 07:55:29.649565 7160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
uuid: "7877484fe82e4183b76bd7f1830f7932"
format_stamp: "Formatted at 2025-08-13 07:55:12 on dist-test-slave-dr1z"
I20250813 07:55:29.652372 7160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 07:55:29.720180 7160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 07:55:29.722158 7160 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 07:55:29.722718 7160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 07:55:29.724642 7183 raft_consensus.cc:491] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:29.725127 7183 raft_consensus.cc:513] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:29.726122 7160 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 07:55:29.727816 7183 leader_election.cc:290] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
W20250813 07:55:29.736927 6909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.106.195:39465: connect: Connection refused (error 111)
I20250813 07:55:29.742301 7189 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250813 07:55:29.750837 6909 leader_election.cc:336] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465): Network error: Client connection negotiation failed: client connection to 127.0.106.195:39465: connect: Connection refused (error 111)
I20250813 07:55:29.752336 7160 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 07:55:29.752645 7160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.001s sys 0.007s
I20250813 07:55:29.752965 7160 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 07:55:29.749534 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1" is_pre_election: true
I20250813 07:55:29.760367 7160 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 07:55:29.760699 7160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.004s sys 0.004s
I20250813 07:55:29.761209 7189 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Bootstrap starting.
W20250813 07:55:29.762125 6909 leader_election.cc:343] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467): Illegal state: must be running to vote when last-logged opid is not known
I20250813 07:55:29.762645 6909 leader_election.cc:304] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d02ee4f03a91457d9f197c6da4ef75b0; no voters: 491f917a421b40c7816aa138436918e1, 7877484fe82e4183b76bd7f1830f7932
I20250813 07:55:29.763470 7183 raft_consensus.cc:2747] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250813 07:55:29.959755 7160 rpc_server.cc:307] RPC server started. Bound to: 127.0.106.195:39465
I20250813 07:55:29.960001 7296 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.106.195:39465 every 8 connection(s)
I20250813 07:55:29.964022 7160 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb
I20250813 07:55:29.967110 427 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu as pid 7160
I20250813 07:55:30.054678 7297 heartbeater.cc:344] Connected to a master server at 127.0.106.254:36111
I20250813 07:55:30.055374 7297 heartbeater.cc:461] Registering TS with master...
I20250813 07:55:30.056972 7297 heartbeater.cc:507] Master 127.0.106.254:36111 requested a full tablet report, sending...
I20250813 07:55:30.063043 6832 ts_manager.cc:194] Registered new tserver with Master: 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:30.070044 427 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 07:55:30.075609 6832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.106.195:53545
I20250813 07:55:30.079741 7189 log.cc:826] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Log is configured to *not* fsync() on all Append() calls
I20250813 07:55:30.312601 7049 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:55:30.313356 7049 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Bootstrap complete.
I20250813 07:55:30.314558 7049 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Time spent bootstrapping tablet: real 2.790s user 2.726s sys 0.040s
I20250813 07:55:30.319639 7049 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:30.321538 7049 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491f917a421b40c7816aa138436918e1, State: Initialized, Role: FOLLOWER
I20250813 07:55:30.322160 7049 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:30.324990 7049 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20250813 07:55:31.081354 7297 heartbeater.cc:499] Master 127.0.106.254:36111 was elected leader, sending a full tablet report...
I20250813 07:55:31.774127 7310 raft_consensus.cc:491] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:31.774490 7310 raft_consensus.cc:513] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:31.775830 7310 leader_election.cc:290] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:31.776793 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1" is_pre_election: true
I20250813 07:55:31.777568 7111 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 1.
I20250813 07:55:31.780145 6909 leader_election.cc:304] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:31.781023 7310 raft_consensus.cc:2802] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250813 07:55:31.781344 7310 raft_consensus.cc:491] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:55:31.781677 7310 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:55:31.789615 7310 raft_consensus.cc:513] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:31.791350 7310 leader_election.cc:290] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 election: Requested vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:31.792203 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1"
I20250813 07:55:31.792753 7111 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:55:31.790833 7252 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932" is_pre_election: true
I20250813 07:55:31.793318 7251 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1b03be001ebc40fdaff20ae647fa957d" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932"
I20250813 07:55:31.801746 7111 raft_consensus.cc:2466] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 2.
I20250813 07:55:31.803115 6909 leader_election.cc:304] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:31.803864 7310 raft_consensus.cc:2802] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 FOLLOWER]: Leader election won for term 2
W20250813 07:55:31.803933 6909 leader_election.cc:343] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465): Illegal state: must be running to vote when last-logged opid is not known
I20250813 07:55:31.805395 7310 raft_consensus.cc:695] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 LEADER]: Becoming Leader. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Running, Role: LEADER
I20250813 07:55:31.806362 7310 consensus_queue.cc:237] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:31.818691 6832 catalog_manager.cc:5582] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 reported cstate change: term changed from 0 to 2, leader changed from <none> to d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193), VOTER 491f917a421b40c7816aa138436918e1 (127.0.106.194) added, VOTER 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195) added, VOTER d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193) added. New cstate: current_term: 2 leader_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20250813 07:55:32.015368 7189 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 07:55:32.016191 7189 tablet_bootstrap.cc:492] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Bootstrap complete.
I20250813 07:55:32.017302 7189 ts_tablet_manager.cc:1397] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Time spent bootstrapping tablet: real 2.257s user 2.187s sys 0.060s
I20250813 07:55:32.022476 7189 raft_consensus.cc:357] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:32.024410 7189 raft_consensus.cc:738] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877484fe82e4183b76bd7f1830f7932, State: Initialized, Role: FOLLOWER
I20250813 07:55:32.025152 7189 consensus_queue.cc:260] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } }
I20250813 07:55:32.027525 7189 ts_tablet_manager.cc:1428] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250813 07:55:32.280830 7111 raft_consensus.cc:1273] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250813 07:55:32.282322 7310 consensus_queue.cc:1035] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250813 07:55:32.307845 7252 raft_consensus.cc:3058] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Advancing to term 2
I20250813 07:55:32.317270 7252 raft_consensus.cc:1273] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 2 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250813 07:55:32.327175 7313 consensus_queue.cc:1035] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250813 07:55:32.353516 6975 consensus_queue.cc:237] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:32.358151 7111 raft_consensus.cc:1273] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250813 07:55:32.361018 7313 consensus_queue.cc:1035] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250813 07:55:32.368126 7310 raft_consensus.cc:2953] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } }
I20250813 07:55:32.373008 7111 raft_consensus.cc:2953] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } }
I20250813 07:55:32.380860 6818 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1b03be001ebc40fdaff20ae647fa957d with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 07:55:32.386412 6832 catalog_manager.cc:5582] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 reported cstate change: config changed from index -1 to 207, VOTER 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195) evicted. New cstate: current_term: 2 leader_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } health_report { overall_health: HEALTHY } } }
I20250813 07:55:32.404827 6975 consensus_queue.cc:237] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:32.409080 7310 raft_consensus.cc:2953] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 491f917a421b40c7816aa138436918e1 (127.0.106.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } } }
I20250813 07:55:32.418234 6818 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1b03be001ebc40fdaff20ae647fa957d with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 07:55:32.422911 6832 catalog_manager.cc:5582] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 reported cstate change: config changed from index 207 to 208, VOTER 491f917a421b40c7816aa138436918e1 (127.0.106.194) evicted. New cstate: current_term: 2 leader_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } health_report { overall_health: HEALTHY } } }
I20250813 07:55:32.421430 7232 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b03be001ebc40fdaff20ae647fa957d with delete_type TABLET_DATA_TOMBSTONED (TS 7877484fe82e4183b76bd7f1830f7932 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:50568
I20250813 07:55:32.430395 7335 tablet_replica.cc:331] stopping tablet replica
I20250813 07:55:32.431227 7335 raft_consensus.cc:2241] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 07:55:32.431871 7335 raft_consensus.cc:2270] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 07:55:32.446606 7091 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b03be001ebc40fdaff20ae647fa957d with delete_type TABLET_DATA_TOMBSTONED (TS 491f917a421b40c7816aa138436918e1 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:40992
I20250813 07:55:32.457221 7337 tablet_replica.cc:331] stopping tablet replica
I20250813 07:55:32.458197 7337 raft_consensus.cc:2241] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 07:55:32.458887 7337 raft_consensus.cc:2270] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 07:55:32.460740 7335 ts_tablet_manager.cc:1905] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 07:55:32.475975 7335 ts_tablet_manager.cc:1918] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250813 07:55:32.476464 7335 log.cc:1199] T 1b03be001ebc40fdaff20ae647fa957d P 7877484fe82e4183b76bd7f1830f7932: Deleting WAL directory at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/wal/wals/1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:32.478389 6818 catalog_manager.cc:4928] TS 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465): tablet 1b03be001ebc40fdaff20ae647fa957d (table pre_rebuild [id=b2748bdd06414a8394a50b8b49ed7370]) successfully deleted
I20250813 07:55:32.484934 7337 ts_tablet_manager.cc:1905] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 07:55:32.500422 7337 ts_tablet_manager.cc:1918] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250813 07:55:32.500892 7337 log.cc:1199] T 1b03be001ebc40fdaff20ae647fa957d P 491f917a421b40c7816aa138436918e1: Deleting WAL directory at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/wal/wals/1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:32.502657 6818 catalog_manager.cc:4928] TS 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467): tablet 1b03be001ebc40fdaff20ae647fa957d (table pre_rebuild [id=b2748bdd06414a8394a50b8b49ed7370]) successfully deleted
W20250813 07:55:32.542591 427 scanner-internal.cc:458] Time spent opening tablet: real 2.413s user 0.007s sys 0.000s
W20250813 07:55:32.657038 6863 debug-util.cc:398] Leaking SignalData structure 0x7b080006f5a0 after lost signal to thread 6800
W20250813 07:55:32.657733 6863 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1920 after lost signal to thread 6866
I20250813 07:55:33.223496 7091 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:55:33.244199 7232 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:55:33.252084 6955 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
27737bcae690471797cea55dfb8fe534 | 127.0.106.254:36111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:38827 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
491f917a421b40c7816aa138436918e1 | 127.0.106.194:34467 | HEALTHY | <none> | 0 | 0
7877484fe82e4183b76bd7f1830f7932 | 127.0.106.195:39465 | HEALTHY | <none> | 0 | 0
d02ee4f03a91457d9f197c6da4ef75b0 | 127.0.106.193:36963 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:36963
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:34467
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:39465
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.193:36963
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.194:34467
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.195:39465
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:38827 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 07:55:33.602948 427 log_verifier.cc:126] Checking tablet 1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:34.017746 427 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:34.020779 6831 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39906:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250813 07:55:34.021559 6831 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39906:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250813 07:55:34.038650 6831 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 27737bcae690471797cea55dfb8fe534: Sending DeleteTablet for 1 replicas of tablet 1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:34.040707 6955 tablet_service.cc:1515] Processing DeleteTablet for tablet 1b03be001ebc40fdaff20ae647fa957d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-13 07:55:34 UTC) from {username='slave'} at 127.0.0.1:51196
I20250813 07:55:34.040867 427 test_util.cc:276] Using random seed: 2068007178
I20250813 07:55:34.053638 7369 tablet_replica.cc:331] stopping tablet replica
I20250813 07:55:34.054641 7369 raft_consensus.cc:2241] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 LEADER]: Raft consensus shutting down.
I20250813 07:55:34.055454 7369 raft_consensus.cc:2270] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 07:55:34.092088 7369 ts_tablet_manager.cc:1905] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250813 07:55:34.107873 7369 ts_tablet_manager.cc:1918] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250813 07:55:34.108434 7369 log.cc:1199] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Deleting WAL directory at /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/wal/wals/1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:34.109357 7369 ts_tablet_manager.cc:1939] T 1b03be001ebc40fdaff20ae647fa957d P d02ee4f03a91457d9f197c6da4ef75b0: Deleting consensus metadata
I20250813 07:55:34.112515 6818 catalog_manager.cc:4928] TS d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193:36963): tablet 1b03be001ebc40fdaff20ae647fa957d (table pre_rebuild [id=b2748bdd06414a8394a50b8b49ed7370]) successfully deleted
I20250813 07:55:34.133093 6831 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39928:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250813 07:55:34.136897 6831 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 07:55:34.164790 7091 tablet_service.cc:1468] Processing CreateTablet for tablet 9310d7ea56b0404f8583e9977b3c614f (DEFAULT_TABLE table=post_rebuild [id=dd0415503ad3476a810271807fc83943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:34.166199 7091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9310d7ea56b0404f8583e9977b3c614f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:34.166076 7232 tablet_service.cc:1468] Processing CreateTablet for tablet 9310d7ea56b0404f8583e9977b3c614f (DEFAULT_TABLE table=post_rebuild [id=dd0415503ad3476a810271807fc83943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:34.167301 7232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9310d7ea56b0404f8583e9977b3c614f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:34.171720 6955 tablet_service.cc:1468] Processing CreateTablet for tablet 9310d7ea56b0404f8583e9977b3c614f (DEFAULT_TABLE table=post_rebuild [id=dd0415503ad3476a810271807fc83943]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 07:55:34.173123 6955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9310d7ea56b0404f8583e9977b3c614f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 07:55:34.187090 7376 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: Bootstrap starting.
I20250813 07:55:34.193836 7376 tablet_bootstrap.cc:654] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:34.220543 7379 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: Bootstrap starting.
I20250813 07:55:34.225145 7378 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: Bootstrap starting.
I20250813 07:55:34.226821 7376 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: No bootstrap required, opened a new log
I20250813 07:55:34.227285 7376 ts_tablet_manager.cc:1397] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.025s
I20250813 07:55:34.229952 7376 raft_consensus.cc:357] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.230643 7376 raft_consensus.cc:383] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:34.230962 7376 raft_consensus.cc:738] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491f917a421b40c7816aa138436918e1, State: Initialized, Role: FOLLOWER
I20250813 07:55:34.233121 7376 consensus_queue.cc:260] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.235803 7378 tablet_bootstrap.cc:654] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:34.239024 7379 tablet_bootstrap.cc:654] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: Neither blocks nor log segments found. Creating new log.
I20250813 07:55:34.251945 7378 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: No bootstrap required, opened a new log
I20250813 07:55:34.253921 7378 ts_tablet_manager.cc:1397] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.004s
I20250813 07:55:34.254621 7376 ts_tablet_manager.cc:1428] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: Time spent starting tablet: real 0.027s user 0.021s sys 0.003s
I20250813 07:55:34.257227 7379 tablet_bootstrap.cc:492] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: No bootstrap required, opened a new log
I20250813 07:55:34.257691 7379 ts_tablet_manager.cc:1397] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20250813 07:55:34.261497 7379 raft_consensus.cc:357] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.262210 7379 raft_consensus.cc:383] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:34.262511 7379 raft_consensus.cc:738] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Initialized, Role: FOLLOWER
I20250813 07:55:34.263244 7379 consensus_queue.cc:260] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.267235 7378 raft_consensus.cc:357] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.268041 7378 raft_consensus.cc:383] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 07:55:34.270140 7378 raft_consensus.cc:738] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7877484fe82e4183b76bd7f1830f7932, State: Initialized, Role: FOLLOWER
I20250813 07:55:34.270990 7378 consensus_queue.cc:260] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
W20250813 07:55:34.274930 7157 tablet.cc:2378] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:55:34.275892 7379 ts_tablet_manager.cc:1428] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
I20250813 07:55:34.289536 7378 ts_tablet_manager.cc:1428] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: Time spent starting tablet: real 0.035s user 0.005s sys 0.014s
W20250813 07:55:34.446074 7021 tablet.cc:2378] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:55:34.448411 7380 raft_consensus.cc:491] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:34.448873 7380 raft_consensus.cc:513] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.449313 7384 raft_consensus.cc:491] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:34.449817 7384 raft_consensus.cc:513] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.454967 7380 leader_election.cc:290] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465), d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193:36963)
I20250813 07:55:34.462497 7383 raft_consensus.cc:491] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 07:55:34.463160 7383 raft_consensus.cc:513] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.465445 7383 leader_election.cc:290] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:34.467128 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1" is_pre_election: true
I20250813 07:55:34.467794 7111 raft_consensus.cc:2466] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 0.
I20250813 07:55:34.469349 6909 leader_election.cc:304] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:34.470422 7383 raft_consensus.cc:2802] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 07:55:34.470827 7383 raft_consensus.cc:491] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 07:55:34.471315 7383 raft_consensus.cc:3058] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:34.491290 7384 leader_election.cc:290] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193:36963)
I20250813 07:55:34.495028 7383 raft_consensus.cc:513] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
I20250813 07:55:34.499632 7383 leader_election.cc:290] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 election: Requested vote from peers 491f917a421b40c7816aa138436918e1 (127.0.106.194:34467), 7877484fe82e4183b76bd7f1830f7932 (127.0.106.195:39465)
I20250813 07:55:34.501075 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1"
I20250813 07:55:34.501682 7111 raft_consensus.cc:3058] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:34.508226 7111 raft_consensus.cc:2466] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 1.
I20250813 07:55:34.509406 6909 leader_election.cc:304] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0; no voters:
I20250813 07:55:34.511250 7383 raft_consensus.cc:2802] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 07:55:34.512419 7252 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932" is_pre_election: true
I20250813 07:55:34.513070 7252 raft_consensus.cc:2466] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 0.
I20250813 07:55:34.515883 7252 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932"
I20250813 07:55:34.516424 7252 raft_consensus.cc:3058] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 0 FOLLOWER]: Advancing to term 1
I20250813 07:55:34.523044 7383 raft_consensus.cc:695] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [term 1 LEADER]: Becoming Leader. State: Replica: d02ee4f03a91457d9f197c6da4ef75b0, State: Running, Role: LEADER
I20250813 07:55:34.523103 7252 raft_consensus.cc:2466] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d02ee4f03a91457d9f197c6da4ef75b0 in term 1.
I20250813 07:55:34.523993 7383 consensus_queue.cc:237] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } }
W20250813 07:55:34.527035 7300 tablet.cc:2378] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 07:55:34.531424 6975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "7877484fe82e4183b76bd7f1830f7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" is_pre_election: true
I20250813 07:55:34.535086 7111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "7877484fe82e4183b76bd7f1830f7932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491f917a421b40c7816aa138436918e1" is_pre_election: true
I20250813 07:55:34.535905 7111 raft_consensus.cc:2391] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7877484fe82e4183b76bd7f1830f7932 in current term 1: Already voted for candidate d02ee4f03a91457d9f197c6da4ef75b0 in this term.
I20250813 07:55:34.537122 7181 leader_election.cc:304] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7877484fe82e4183b76bd7f1830f7932; no voters: 491f917a421b40c7816aa138436918e1, d02ee4f03a91457d9f197c6da4ef75b0
I20250813 07:55:34.538661 7384 raft_consensus.cc:2747] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250813 07:55:34.548302 7252 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "491f917a421b40c7816aa138436918e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7877484fe82e4183b76bd7f1830f7932" is_pre_election: true
I20250813 07:55:34.549274 7252 raft_consensus.cc:2391] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 491f917a421b40c7816aa138436918e1 in current term 1: Already voted for candidate d02ee4f03a91457d9f197c6da4ef75b0 in this term.
I20250813 07:55:34.555317 6831 catalog_manager.cc:5582] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d02ee4f03a91457d9f197c6da4ef75b0 (127.0.106.193). New cstate: current_term: 1 leader_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" member_type: VOTER last_known_addr { host: "127.0.106.193" port: 36963 } health_report { overall_health: HEALTHY } } }
I20250813 07:55:34.558887 6975 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9310d7ea56b0404f8583e9977b3c614f" candidate_uuid: "491f917a421b40c7816aa138436918e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d02ee4f03a91457d9f197c6da4ef75b0" is_pre_election: true
I20250813 07:55:34.560518 7045 leader_election.cc:304] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 491f917a421b40c7816aa138436918e1; no voters: 7877484fe82e4183b76bd7f1830f7932, d02ee4f03a91457d9f197c6da4ef75b0
I20250813 07:55:34.561720 7380 raft_consensus.cc:2747] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250813 07:55:34.748448 7252 raft_consensus.cc:1273] T 9310d7ea56b0404f8583e9977b3c614f P 7877484fe82e4183b76bd7f1830f7932 [term 1 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 07:55:34.749959 7383 consensus_queue.cc:1035] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7877484fe82e4183b76bd7f1830f7932" member_type: VOTER last_known_addr { host: "127.0.106.195" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:55:34.760293 7111 raft_consensus.cc:1273] T 9310d7ea56b0404f8583e9977b3c614f P 491f917a421b40c7816aa138436918e1 [term 1 FOLLOWER]: Refusing update from remote peer d02ee4f03a91457d9f197c6da4ef75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 07:55:34.762065 7383 consensus_queue.cc:1035] T 9310d7ea56b0404f8583e9977b3c614f P d02ee4f03a91457d9f197c6da4ef75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "491f917a421b40c7816aa138436918e1" member_type: VOTER last_known_addr { host: "127.0.106.194" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 07:55:34.798355 7402 mvcc.cc:204] Tried to move back new op lower bound from 7188773825508536320 to 7188773824618786816. Current Snapshot: MvccSnapshot[applied={T|T < 7188773825508536320}]
I20250813 07:55:34.822794 7403 mvcc.cc:204] Tried to move back new op lower bound from 7188773825508536320 to 7188773824618786816. Current Snapshot: MvccSnapshot[applied={T|T < 7188773825508536320}]
I20250813 07:55:34.846839 7404 mvcc.cc:204] Tried to move back new op lower bound from 7188773825508536320 to 7188773824618786816. Current Snapshot: MvccSnapshot[applied={T|T < 7188773825508536320}]
W20250813 07:55:37.091806 7399 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.000s sys 0.000s
I20250813 07:55:39.701608 7091 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:55:39.710348 7232 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 07:55:39.714200 6955 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
27737bcae690471797cea55dfb8fe534 | 127.0.106.254:36111 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:38827 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
491f917a421b40c7816aa138436918e1 | 127.0.106.194:34467 | HEALTHY | <none> | 0 | 0
7877484fe82e4183b76bd7f1830f7932 | 127.0.106.195:39465 | HEALTHY | <none> | 0 | 0
d02ee4f03a91457d9f197c6da4ef75b0 | 127.0.106.193:36963 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.106.193 | experimental | 127.0.106.193:36963
local_ip_for_outbound_sockets | 127.0.106.194 | experimental | 127.0.106.194:34467
local_ip_for_outbound_sockets | 127.0.106.195 | experimental | 127.0.106.195:39465
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-0/data/info.pb | hidden | 127.0.106.193:36963
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-1/data/info.pb | hidden | 127.0.106.194:34467
server_dump_info_path | /tmp/dist-test-taskYtHc8N/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755071556474372-427-0/minicluster-data/ts-2/data/info.pb | hidden | 127.0.106.195:39465
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.0.106.212:38827 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 07:55:39.925765 427 log_verifier.cc:126] Checking tablet 1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:39.926083 427 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 1b03be001ebc40fdaff20ae647fa957d
I20250813 07:55:39.926355 427 log_verifier.cc:126] Checking tablet 9310d7ea56b0404f8583e9977b3c614f
I20250813 07:55:40.694312 427 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 9310d7ea56b0404f8583e9977b3c614f
I20250813 07:55:40.719899 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6869
I20250813 07:55:40.765936 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 7024
I20250813 07:55:40.797960 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 7160
I20250813 07:55:40.827850 427 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYtHc8N/build/tsan/bin/kudu with pid 6799
2025-08-13T07:55:40Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36841 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36841 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (184338 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST
I20250813 07:55:40.891218 427 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 4 messages since previous log ~50 seconds ago