Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 20:09:23.365929 18066 test_util.cc:276] Using random seed: -1411516500
W20250622 20:09:24.581424 18066 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.173s user 0.478s sys 0.695s
W20250622 20:09:24.581790 18066 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.174s user 0.478s sys 0.695s
I20250622 20:09:24.584257 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:09:24.584470 18066 ts_itest-base.cc:116] --------------
I20250622 20:09:24.584651 18066 ts_itest-base.cc:117] 4 tablet servers
I20250622 20:09:24.584849 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:09:24.585026 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:09:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:09:24Z Disabled control of system clock
I20250622 20:09:24.621497 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:40581
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:40581 with env {}
W20250622 20:09:24.924530 18080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:24.925109 18080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:24.925509 18080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:24.956838 18080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:09:24.957147 18080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:24.957343 18080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:09:24.957522 18080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:09:24.992151 18080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:40581
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:24.993592 18080 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:24.995267 18080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:25.010330 18087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:25.010376 18089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:25.010617 18086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:25.011801 18080 server_base.cc:1048] running on GCE node
I20250622 20:09:26.164561 18080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:26.167733 18080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:26.169162 18080 hybrid_clock.cc:648] HybridClock initialized: now 1750622966169118 us; error 59 us; skew 500 ppm
I20250622 20:09:26.169936 18080 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:26.176805 18080 webserver.cc:469] Webserver started at http://127.17.164.190:38509/ using document root <none> and password file <none>
I20250622 20:09:26.177690 18080 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:26.177865 18080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:26.178287 18080 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:26.182624 18080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bb83152251914e0b9830cf8cb89f27be"
format_stamp: "Formatted at 2025-06-22 20:09:26 on dist-test-slave-9mkf"
I20250622 20:09:26.183635 18080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bb83152251914e0b9830cf8cb89f27be"
format_stamp: "Formatted at 2025-06-22 20:09:26 on dist-test-slave-9mkf"
I20250622 20:09:26.190848 18080 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250622 20:09:26.196182 18096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:26.197182 18080 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:09:26.197513 18080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bb83152251914e0b9830cf8cb89f27be"
format_stamp: "Formatted at 2025-06-22 20:09:26 on dist-test-slave-9mkf"
I20250622 20:09:26.197813 18080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:26.262323 18080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:26.263736 18080 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:26.264185 18080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:26.332613 18080 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:40581
I20250622 20:09:26.332676 18147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:40581 every 8 connection(s)
I20250622 20:09:26.335354 18080 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:09:26.339779 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18080
I20250622 20:09:26.340332 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:09:26.340680 18148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:26.360495 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Bootstrap starting.
I20250622 20:09:26.365922 18148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:26.367538 18148 log.cc:826] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:26.372475 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: No bootstrap required, opened a new log
I20250622 20:09:26.390781 18148 raft_consensus.cc:357] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:26.391505 18148 raft_consensus.cc:383] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:26.391717 18148 raft_consensus.cc:738] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb83152251914e0b9830cf8cb89f27be, State: Initialized, Role: FOLLOWER
I20250622 20:09:26.392341 18148 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:26.392927 18148 raft_consensus.cc:397] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:09:26.393294 18148 raft_consensus.cc:491] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:09:26.393597 18148 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:26.397538 18148 raft_consensus.cc:513] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:26.398293 18148 leader_election.cc:304] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb83152251914e0b9830cf8cb89f27be; no voters:
I20250622 20:09:26.399785 18148 leader_election.cc:290] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:09:26.400439 18153 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:09:26.402674 18153 raft_consensus.cc:695] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 LEADER]: Becoming Leader. State: Replica: bb83152251914e0b9830cf8cb89f27be, State: Running, Role: LEADER
I20250622 20:09:26.403452 18153 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:26.403921 18148 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:09:26.414099 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb83152251914e0b9830cf8cb89f27be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } } }
I20250622 20:09:26.414765 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:26.416150 18155 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb83152251914e0b9830cf8cb89f27be. Latest consensus state: current_term: 1 leader_uuid: "bb83152251914e0b9830cf8cb89f27be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } } }
I20250622 20:09:26.416918 18155 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:26.417447 18162 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:09:26.428773 18162 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:09:26.443475 18162 catalog_manager.cc:1349] Generated new cluster ID: ec0ba42fba9c4285a2cb4afea60a0e1b
I20250622 20:09:26.443722 18162 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:09:26.468770 18162 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:09:26.470230 18162 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:09:26.494052 18162 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Generated new TSK 0
I20250622 20:09:26.494894 18162 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:09:26.513608 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 20:09:26.830142 18172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:26.830641 18172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:26.831110 18172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:26.862236 18172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:26.863078 18172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:09:26.898873 18172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:26.900230 18172 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:26.901911 18172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:26.919098 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:26.920464 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:26.921548 18172 server_base.cc:1048] running on GCE node
W20250622 20:09:26.921547 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:28.098508 18172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:28.102535 18172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:28.104032 18172 hybrid_clock.cc:648] HybridClock initialized: now 1750622968103977 us; error 74 us; skew 500 ppm
I20250622 20:09:28.104959 18172 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:28.112170 18172 webserver.cc:469] Webserver started at http://127.17.164.129:46703/ using document root <none> and password file <none>
I20250622 20:09:28.113183 18172 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:28.113415 18172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:28.113878 18172 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:28.118367 18172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b27eab7a62c04658a0dcde9364054f20"
format_stamp: "Formatted at 2025-06-22 20:09:28 on dist-test-slave-9mkf"
I20250622 20:09:28.119446 18172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b27eab7a62c04658a0dcde9364054f20"
format_stamp: "Formatted at 2025-06-22 20:09:28 on dist-test-slave-9mkf"
I20250622 20:09:28.126782 18172 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250622 20:09:28.132841 18188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:28.133975 18172 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 20:09:28.134277 18172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b27eab7a62c04658a0dcde9364054f20"
format_stamp: "Formatted at 2025-06-22 20:09:28 on dist-test-slave-9mkf"
I20250622 20:09:28.134613 18172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:28.189386 18172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:28.190845 18172 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:28.191350 18172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:28.194152 18172 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:28.198976 18172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:28.199201 18172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:28.199476 18172 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:28.199640 18172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:28.344558 18172 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:40463
I20250622 20:09:28.344756 18300 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:40463 every 8 connection(s)
I20250622 20:09:28.347209 18172 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:09:28.348150 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18172
I20250622 20:09:28.348672 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:09:28.356668 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:09:28.373926 18301 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:28.374349 18301 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:28.375344 18301 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:28.378815 18113 ts_manager.cc:194] Registered new tserver with Master: b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463)
I20250622 20:09:28.381812 18113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:58303
W20250622 20:09:28.664494 18305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:28.664968 18305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:28.665400 18305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:28.698076 18305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:28.698871 18305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:09:28.732820 18305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:28.734102 18305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:28.735642 18305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:28.751662 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:29.385334 18301 heartbeater.cc:499] Master 127.17.164.190:40581 was elected leader, sending a full tablet report...
W20250622 20:09:28.753114 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:28.755106 18305 server_base.cc:1048] running on GCE node
W20250622 20:09:28.755000 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:29.893283 18305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:29.895386 18305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:29.896785 18305 hybrid_clock.cc:648] HybridClock initialized: now 1750622969896747 us; error 57 us; skew 500 ppm
I20250622 20:09:29.897575 18305 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:29.907248 18305 webserver.cc:469] Webserver started at http://127.17.164.130:39811/ using document root <none> and password file <none>
I20250622 20:09:29.908123 18305 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:29.908330 18305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:29.908835 18305 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:29.912993 18305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1baa44b58f1546ef8bb0fd5072896b8b"
format_stamp: "Formatted at 2025-06-22 20:09:29 on dist-test-slave-9mkf"
I20250622 20:09:29.914390 18305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1baa44b58f1546ef8bb0fd5072896b8b"
format_stamp: "Formatted at 2025-06-22 20:09:29 on dist-test-slave-9mkf"
I20250622 20:09:29.921434 18305 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250622 20:09:29.926438 18321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:29.927390 18305 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250622 20:09:29.927691 18305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1baa44b58f1546ef8bb0fd5072896b8b"
format_stamp: "Formatted at 2025-06-22 20:09:29 on dist-test-slave-9mkf"
I20250622 20:09:29.928010 18305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:29.986752 18305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:29.988165 18305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:29.989177 18305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:29.992581 18305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:29.999148 18305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:29.999459 18305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:29.999796 18305 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:30.000025 18305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:30.156201 18305 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:35793
I20250622 20:09:30.156296 18433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:35793 every 8 connection(s)
I20250622 20:09:30.158496 18305 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:09:30.167855 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18305
I20250622 20:09:30.168357 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:09:30.174451 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:09:30.182448 18434 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:30.182838 18434 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:30.183782 18434 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:30.185734 18113 ts_manager.cc:194] Registered new tserver with Master: 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130:35793)
I20250622 20:09:30.187268 18113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:38209
W20250622 20:09:30.470757 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:30.471251 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:30.471753 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:30.501471 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:30.502300 18438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:09:30.534994 18438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:30.536226 18438 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:30.537734 18438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:30.553761 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:31.190523 18434 heartbeater.cc:499] Master 127.17.164.190:40581 was elected leader, sending a full tablet report...
W20250622 20:09:30.554610 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:31.712935 18446 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250622 20:09:31.713913 18438 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.164s user 0.461s sys 0.702s
W20250622 20:09:31.714337 18438 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.165s user 0.461s sys 0.702s
W20250622 20:09:31.714483 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:31.714632 18438 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:09:31.716138 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:31.718905 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:31.720399 18438 hybrid_clock.cc:648] HybridClock initialized: now 1750622971720360 us; error 30 us; skew 500 ppm
I20250622 20:09:31.721601 18438 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:31.731549 18438 webserver.cc:469] Webserver started at http://127.17.164.131:44341/ using document root <none> and password file <none>
I20250622 20:09:31.732998 18438 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:31.733351 18438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:31.733978 18438 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:31.741371 18438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d5052250ec6041bfb6a01fe4ba815b43"
format_stamp: "Formatted at 2025-06-22 20:09:31 on dist-test-slave-9mkf"
I20250622 20:09:31.743018 18438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d5052250ec6041bfb6a01fe4ba815b43"
format_stamp: "Formatted at 2025-06-22 20:09:31 on dist-test-slave-9mkf"
I20250622 20:09:31.753453 18438 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.011s sys 0.000s
I20250622 20:09:31.761821 18454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:31.763077 18438 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250622 20:09:31.763470 18438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d5052250ec6041bfb6a01fe4ba815b43"
format_stamp: "Formatted at 2025-06-22 20:09:31 on dist-test-slave-9mkf"
I20250622 20:09:31.763912 18438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:31.844514 18438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:31.845963 18438 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:31.846318 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:31.849112 18438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:31.853346 18438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:31.853538 18438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:31.853747 18438 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:31.853878 18438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:32.003938 18438 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:38107
I20250622 20:09:32.004024 18566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:38107 every 8 connection(s)
I20250622 20:09:32.006446 18438 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:09:32.011165 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18438
I20250622 20:09:32.011574 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:09:32.017658 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.132:0
--local_ip_for_outbound_sockets=127.17.164.132
--webserver_interface=127.17.164.132
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:09:32.030475 18567 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:32.030941 18567 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:32.032284 18567 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:32.035421 18113 ts_manager.cc:194] Registered new tserver with Master: d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107)
I20250622 20:09:32.037245 18113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:51637
W20250622 20:09:32.319638 18571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:32.320178 18571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:32.321008 18571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:32.360908 18571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:32.362083 18571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.132
I20250622 20:09:32.396164 18571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.164.132
--webserver_port=0
--tserver_master_addrs=127.17.164.190:40581
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.132
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:32.397483 18571 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:32.399547 18571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:32.416937 18577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:33.041007 18567 heartbeater.cc:499] Master 127.17.164.190:40581 was elected leader, sending a full tablet report...
W20250622 20:09:32.417004 18580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:32.417912 18578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:32.419800 18571 server_base.cc:1048] running on GCE node
I20250622 20:09:33.595680 18571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:33.597837 18571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:33.599154 18571 hybrid_clock.cc:648] HybridClock initialized: now 1750622973599136 us; error 41 us; skew 500 ppm
I20250622 20:09:33.599931 18571 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:33.606710 18571 webserver.cc:469] Webserver started at http://127.17.164.132:36981/ using document root <none> and password file <none>
I20250622 20:09:33.607582 18571 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:33.607780 18571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:33.608203 18571 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:33.612538 18571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4feef58d9e424d2899e5f8871a521526"
format_stamp: "Formatted at 2025-06-22 20:09:33 on dist-test-slave-9mkf"
I20250622 20:09:33.613641 18571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4feef58d9e424d2899e5f8871a521526"
format_stamp: "Formatted at 2025-06-22 20:09:33 on dist-test-slave-9mkf"
I20250622 20:09:33.620853 18571 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 20:09:33.626348 18588 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:33.627230 18571 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250622 20:09:33.627518 18571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4feef58d9e424d2899e5f8871a521526"
format_stamp: "Formatted at 2025-06-22 20:09:33 on dist-test-slave-9mkf"
I20250622 20:09:33.627806 18571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:33.679639 18571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:33.681026 18571 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:33.681453 18571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:33.683800 18571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:33.687724 18571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:33.687930 18571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:33.688148 18571 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:33.688316 18571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:33.835975 18571 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.132:34397
I20250622 20:09:33.836071 18700 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.132:34397 every 8 connection(s)
I20250622 20:09:33.838327 18571 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 20:09:33.840020 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18571
I20250622 20:09:33.840843 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 20:09:33.861104 18701 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:33.861620 18701 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:33.862709 18701 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:33.864490 18113 ts_manager.cc:194] Registered new tserver with Master: 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397)
I20250622 20:09:33.865676 18113 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.132:37341
I20250622 20:09:33.876319 18066 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 20:09:33.915849 18113 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54276:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 20:09:33.985252 18636 tablet_service.cc:1468] Processing CreateTablet for tablet a66e023fcc594ca8a3267279f1ccba37 (DEFAULT_TABLE table=TestTable [id=3e078579a3d640cd8a577735f4c54e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:33.987633 18636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66e023fcc594ca8a3267279f1ccba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:33.988837 18502 tablet_service.cc:1468] Processing CreateTablet for tablet a66e023fcc594ca8a3267279f1ccba37 (DEFAULT_TABLE table=TestTable [id=3e078579a3d640cd8a577735f4c54e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:33.990589 18502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66e023fcc594ca8a3267279f1ccba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:33.992982 18236 tablet_service.cc:1468] Processing CreateTablet for tablet a66e023fcc594ca8a3267279f1ccba37 (DEFAULT_TABLE table=TestTable [id=3e078579a3d640cd8a577735f4c54e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:33.994879 18236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66e023fcc594ca8a3267279f1ccba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:34.022734 18721 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Bootstrap starting.
I20250622 20:09:34.025588 18722 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Bootstrap starting.
I20250622 20:09:34.026398 18720 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Bootstrap starting.
I20250622 20:09:34.031271 18721 tablet_bootstrap.cc:654] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:34.033996 18721 log.cc:826] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:34.034943 18722 tablet_bootstrap.cc:654] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:34.038275 18722 log.cc:826] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:34.040756 18720 tablet_bootstrap.cc:654] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:34.043219 18720 log.cc:826] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:34.045138 18722 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: No bootstrap required, opened a new log
I20250622 20:09:34.045593 18721 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: No bootstrap required, opened a new log
I20250622 20:09:34.045643 18722 ts_tablet_manager.cc:1397] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20250622 20:09:34.046098 18721 ts_tablet_manager.cc:1397] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.014s
I20250622 20:09:34.060897 18720 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: No bootstrap required, opened a new log
I20250622 20:09:34.061595 18720 ts_tablet_manager.cc:1397] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Time spent bootstrapping tablet: real 0.036s user 0.021s sys 0.000s
I20250622 20:09:34.069860 18722 raft_consensus.cc:357] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.070500 18722 raft_consensus.cc:383] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:34.070768 18722 raft_consensus.cc:738] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27eab7a62c04658a0dcde9364054f20, State: Initialized, Role: FOLLOWER
I20250622 20:09:34.071480 18722 consensus_queue.cc:260] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.073946 18721 raft_consensus.cc:357] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.074941 18721 raft_consensus.cc:383] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:34.075279 18721 raft_consensus.cc:738] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4feef58d9e424d2899e5f8871a521526, State: Initialized, Role: FOLLOWER
I20250622 20:09:34.076071 18722 ts_tablet_manager.cc:1428] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Time spent starting tablet: real 0.030s user 0.019s sys 0.008s
I20250622 20:09:34.076797 18721 consensus_queue.cc:260] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.080600 18701 heartbeater.cc:499] Master 127.17.164.190:40581 was elected leader, sending a full tablet report...
I20250622 20:09:34.082021 18721 ts_tablet_manager.cc:1428] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Time spent starting tablet: real 0.036s user 0.025s sys 0.009s
I20250622 20:09:34.088842 18720 raft_consensus.cc:357] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.089707 18720 raft_consensus.cc:383] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:34.089995 18720 raft_consensus.cc:738] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5052250ec6041bfb6a01fe4ba815b43, State: Initialized, Role: FOLLOWER
I20250622 20:09:34.090909 18720 consensus_queue.cc:260] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
W20250622 20:09:34.093206 18702 tablet.cc:2378] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:09:34.094606 18720 ts_tablet_manager.cc:1428] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Time spent starting tablet: real 0.033s user 0.028s sys 0.004s
I20250622 20:09:34.107973 18726 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250622 20:09:34.108815 18302 tablet.cc:2378] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:09:34.108415 18726 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.111119 18726 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397), d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107)
I20250622 20:09:34.122784 18656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "b27eab7a62c04658a0dcde9364054f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4feef58d9e424d2899e5f8871a521526" is_pre_election: true
I20250622 20:09:34.123499 18656 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27eab7a62c04658a0dcde9364054f20 in term 0.
I20250622 20:09:34.124766 18190 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526, b27eab7a62c04658a0dcde9364054f20; no voters:
I20250622 20:09:34.125450 18726 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:09:34.125725 18726 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:09:34.125952 18726 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:34.126155 18522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "b27eab7a62c04658a0dcde9364054f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43" is_pre_election: true
I20250622 20:09:34.126761 18522 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27eab7a62c04658a0dcde9364054f20 in term 0.
I20250622 20:09:34.130426 18726 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.131688 18726 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [CANDIDATE]: Term 1 election: Requested vote from peers 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397), d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107)
I20250622 20:09:34.132375 18656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "b27eab7a62c04658a0dcde9364054f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4feef58d9e424d2899e5f8871a521526"
I20250622 20:09:34.132612 18522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "b27eab7a62c04658a0dcde9364054f20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
I20250622 20:09:34.132884 18656 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:34.133116 18522 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:34.139861 18522 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27eab7a62c04658a0dcde9364054f20 in term 1.
I20250622 20:09:34.140496 18656 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27eab7a62c04658a0dcde9364054f20 in term 1.
I20250622 20:09:34.140821 18190 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b27eab7a62c04658a0dcde9364054f20, d5052250ec6041bfb6a01fe4ba815b43; no voters:
I20250622 20:09:34.141497 18726 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:09:34.142869 18726 raft_consensus.cc:695] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [term 1 LEADER]: Becoming Leader. State: Replica: b27eab7a62c04658a0dcde9364054f20, State: Running, Role: LEADER
I20250622 20:09:34.143687 18726 consensus_queue.cc:237] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:34.154678 18113 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 reported cstate change: term changed from 0 to 1, leader changed from <none> to b27eab7a62c04658a0dcde9364054f20 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "b27eab7a62c04658a0dcde9364054f20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } health_report { overall_health: HEALTHY } } }
I20250622 20:09:34.180807 18066 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250622 20:09:34.183917 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b27eab7a62c04658a0dcde9364054f20 to finish bootstrapping
I20250622 20:09:34.197885 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d5052250ec6041bfb6a01fe4ba815b43 to finish bootstrapping
I20250622 20:09:34.208972 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4feef58d9e424d2899e5f8871a521526 to finish bootstrapping
I20250622 20:09:34.218788 18066 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250622 20:09:34.221813 18066 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "a66e023fcc594ca8a3267279f1ccba37"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "4feef58d9e424d2899e5f8871a521526"
rpc_addresses {
host: "127.17.164.132"
port: 34397
}
}
ts_infos {
permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
rpc_addresses {
host: "127.17.164.131"
port: 38107
}
}
ts_infos {
permanent_uuid: "b27eab7a62c04658a0dcde9364054f20"
rpc_addresses {
host: "127.17.164.129"
port: 40463
}
}
W20250622 20:09:34.263446 18568 tablet.cc:2378] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:09:34.529696 18726 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:09:34.548795 18731 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P b27eab7a62c04658a0dcde9364054f20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 20:09:34.552266 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18172
W20250622 20:09:34.584425 18589 connection.cc:537] server connection from 127.17.164.129:41875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 20:09:34.585211 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18080
W20250622 20:09:34.585193 18100 connection.cc:537] server connection from 127.17.164.129:58303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 20:09:34.615782 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:40581
--webserver_interface=127.17.164.190
--webserver_port=38509
--builtin_ntp_servers=127.17.164.148:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:40581 with env {}
W20250622 20:09:34.924255 18739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:34.924866 18739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:34.925312 18739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:34.956121 18739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:09:34.956449 18739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:34.956746 18739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:09:34.956995 18739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:09:34.991535 18739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:40581
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=38509
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:34.992962 18739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:34.994622 18739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:35.008352 18746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:35.120573 18567 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:40581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:40581: connect: Connection refused (error 111)
W20250622 20:09:35.225042 18434 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:40581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:40581: connect: Connection refused (error 111)
W20250622 20:09:35.573050 18701 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:40581 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:40581: connect: Connection refused (error 111)
I20250622 20:09:36.056502 18754 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b27eab7a62c04658a0dcde9364054f20)
I20250622 20:09:36.057387 18754 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:36.069170 18754 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107), b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463)
W20250622 20:09:36.070330 18590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111)
W20250622 20:09:36.078477 18590 leader_election.cc:336] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463): Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111)
I20250622 20:09:36.091507 18522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "4feef58d9e424d2899e5f8871a521526" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43" is_pre_election: true
I20250622 20:09:36.092284 18522 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4feef58d9e424d2899e5f8871a521526 in term 1.
I20250622 20:09:36.093946 18590 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526, d5052250ec6041bfb6a01fe4ba815b43; no voters: b27eab7a62c04658a0dcde9364054f20
I20250622 20:09:36.095333 18754 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 20:09:36.095702 18754 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b27eab7a62c04658a0dcde9364054f20)
I20250622 20:09:36.096107 18754 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:36.106024 18754 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:36.109722 18522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "4feef58d9e424d2899e5f8871a521526" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
I20250622 20:09:36.110426 18522 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:36.119330 18522 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4feef58d9e424d2899e5f8871a521526 in term 2.
I20250622 20:09:36.123450 18754 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 election: Requested vote from peers d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107), b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463)
I20250622 20:09:36.125571 18590 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526, d5052250ec6041bfb6a01fe4ba815b43; no voters:
W20250622 20:09:36.128111 18590 leader_election.cc:336] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463): Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111)
I20250622 20:09:36.129356 18754 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:09:36.137789 18754 raft_consensus.cc:695] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 LEADER]: Becoming Leader. State: Replica: 4feef58d9e424d2899e5f8871a521526, State: Running, Role: LEADER
I20250622 20:09:36.139286 18754 consensus_queue.cc:237] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:36.572695 18522 raft_consensus.cc:1273] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 2 FOLLOWER]: Refusing update from remote peer 4feef58d9e424d2899e5f8871a521526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 20:09:36.588271 18754 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.005s
W20250622 20:09:36.645287 18590 consensus_peers.cc:487] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 -> Peer b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463): Couldn't send request to peer b27eab7a62c04658a0dcde9364054f20. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 20:09:36.408872 18744 debug-util.cc:398] Leaking SignalData structure 0x7b0800008680 after lost signal to thread 18739
W20250622 20:09:36.693482 18739 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.685s user 0.551s sys 1.084s
W20250622 20:09:36.693806 18739 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.685s user 0.551s sys 1.084s
W20250622 20:09:35.008345 18745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:36.694952 18747 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1685 milliseconds
I20250622 20:09:36.696164 18739 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 20:09:36.696185 18748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:36.699294 18739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:36.701663 18739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:36.703013 18739 hybrid_clock.cc:648] HybridClock initialized: now 1750622976702978 us; error 49 us; skew 500 ppm
I20250622 20:09:36.703773 18739 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:36.716161 18739 webserver.cc:469] Webserver started at http://127.17.164.190:38509/ using document root <none> and password file <none>
I20250622 20:09:36.717119 18739 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:36.717324 18739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:36.724949 18739 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.002s
I20250622 20:09:36.729295 18771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:36.730281 18739 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 20:09:36.730604 18739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bb83152251914e0b9830cf8cb89f27be"
format_stamp: "Formatted at 2025-06-22 20:09:26 on dist-test-slave-9mkf"
I20250622 20:09:36.732383 18739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:36.783056 18739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:36.784404 18739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:36.784873 18739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:36.856037 18739 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:40581
I20250622 20:09:36.856107 18822 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:40581 every 8 connection(s)
I20250622 20:09:36.858755 18739 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:09:36.861972 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18739
I20250622 20:09:36.862576 18066 kudu-admin-test.cc:735] Forcing unsafe config change on tserver d5052250ec6041bfb6a01fe4ba815b43
I20250622 20:09:36.872453 18823 sys_catalog.cc:263] Verifying existing consensus state
I20250622 20:09:36.876858 18823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Bootstrap starting.
I20250622 20:09:36.912617 18823 log.cc:826] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:36.933790 18823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:09:36.934512 18823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Bootstrap complete.
I20250622 20:09:36.954598 18823 raft_consensus.cc:357] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:36.956421 18823 raft_consensus.cc:738] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb83152251914e0b9830cf8cb89f27be, State: Initialized, Role: FOLLOWER
I20250622 20:09:36.957122 18823 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:36.957599 18823 raft_consensus.cc:397] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:09:36.957898 18823 raft_consensus.cc:491] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:09:36.958186 18823 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:36.963632 18823 raft_consensus.cc:513] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:36.964257 18823 leader_election.cc:304] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb83152251914e0b9830cf8cb89f27be; no voters:
I20250622 20:09:36.966347 18823 leader_election.cc:290] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:09:36.966755 18827 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:09:36.969661 18827 raft_consensus.cc:695] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [term 2 LEADER]: Becoming Leader. State: Replica: bb83152251914e0b9830cf8cb89f27be, State: Running, Role: LEADER
I20250622 20:09:36.970445 18827 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } }
I20250622 20:09:36.971318 18823 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:09:36.979780 18828 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb83152251914e0b9830cf8cb89f27be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } } }
I20250622 20:09:36.980441 18828 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:36.982153 18829 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb83152251914e0b9830cf8cb89f27be. Latest consensus state: current_term: 2 leader_uuid: "bb83152251914e0b9830cf8cb89f27be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb83152251914e0b9830cf8cb89f27be" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 40581 } } }
I20250622 20:09:36.982926 18829 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:36.985950 18833 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:09:37.011893 18833 catalog_manager.cc:671] Loaded metadata for table TestTable [id=3e078579a3d640cd8a577735f4c54e38]
I20250622 20:09:37.030947 18833 tablet_loader.cc:96] loaded metadata for tablet a66e023fcc594ca8a3267279f1ccba37 (table TestTable [id=3e078579a3d640cd8a577735f4c54e38])
I20250622 20:09:37.032848 18833 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:09:37.042165 18833 catalog_manager.cc:1261] Loaded cluster ID: ec0ba42fba9c4285a2cb4afea60a0e1b
I20250622 20:09:37.052889 18833 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:09:37.068538 18833 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:09:37.076975 18833 catalog_manager.cc:5966] T 00000000000000000000000000000000 P bb83152251914e0b9830cf8cb89f27be: Loaded TSK: 0
I20250622 20:09:37.087150 18833 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 20:09:37.223201 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:37.223748 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:37.264302 18825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 20:09:37.274200 18434 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:37.279798 18788 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" instance_seqno: 1750622970121394) as {username='slave'} at 127.17.164.130:55005; Asking this server to re-register.
I20250622 20:09:37.281200 18434 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:37.281905 18434 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:37.284462 18788 ts_manager.cc:194] Registered new tserver with Master: 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130:35793)
I20250622 20:09:37.697731 18701 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:37.702957 18788 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4feef58d9e424d2899e5f8871a521526" instance_seqno: 1750622973803185) as {username='slave'} at 127.17.164.132:47167; Asking this server to re-register.
I20250622 20:09:37.705212 18701 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:37.705840 18701 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:37.707536 18567 heartbeater.cc:344] Connected to a master server at 127.17.164.190:40581
I20250622 20:09:37.711395 18788 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" instance_seqno: 1750622971969596) as {username='slave'} at 127.17.164.131:52219; Asking this server to re-register.
I20250622 20:09:37.711869 18787 ts_manager.cc:194] Registered new tserver with Master: 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397)
I20250622 20:09:37.713339 18567 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:37.714272 18567 heartbeater.cc:507] Master 127.17.164.190:40581 requested a full tablet report, sending...
I20250622 20:09:37.715919 18787 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 reported cstate change: term changed from 1 to 2, leader changed from b27eab7a62c04658a0dcde9364054f20 (127.17.164.129) to 4feef58d9e424d2899e5f8871a521526 (127.17.164.132). New cstate: current_term: 2 leader_uuid: "4feef58d9e424d2899e5f8871a521526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } health_report { overall_health: UNKNOWN } } }
I20250622 20:09:37.718165 18788 ts_manager.cc:194] Registered new tserver with Master: d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107)
W20250622 20:09:38.228724 18590 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.164.131:38107, user_credentials={real_user=slave}} blocked reactor thread for 51431.3us
W20250622 20:09:38.713783 18851 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 18825
W20250622 20:09:38.714318 18851 kernel_stack_watchdog.cc:198] Thread 18825 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@3/src/kudu/util/thread.cc:641 for 402ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250622 20:09:38.824965 18825 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.515s user 0.592s sys 0.857s
W20250622 20:09:38.977356 18825 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.667s user 0.593s sys 0.874s
I20250622 20:09:39.022706 18522 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
tablet_id: "a66e023fcc594ca8a3267279f1ccba37"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "4feef58d9e424d2899e5f8871a521526"
}
peers {
permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
}
}
from {username='slave'} at 127.0.0.1:54588
W20250622 20:09:39.023699 18522 raft_consensus.cc:2216] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true
I20250622 20:09:39.024494 18522 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 2 FOLLOWER]: Advancing to term 3
W20250622 20:09:39.034930 18590 consensus_peers.cc:487] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 -> Peer b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463): Couldn't send request to peer b27eab7a62c04658a0dcde9364054f20. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 20:09:39.106256 18522 raft_consensus.cc:1238] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Rejecting Update request from peer 4feef58d9e424d2899e5f8871a521526 for earlier term 2. Current term is 3. Ops: []
I20250622 20:09:39.107486 18862 consensus_queue.cc:1046] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 20:09:39.108963 18877 raft_consensus.cc:3053] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 LEADER]: Stepping down as leader of term 2
I20250622 20:09:39.109191 18877 raft_consensus.cc:738] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4feef58d9e424d2899e5f8871a521526, State: Running, Role: LEADER
I20250622 20:09:39.109817 18877 consensus_queue.cc:260] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:39.110774 18877 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 2 FOLLOWER]: Advancing to term 3
W20250622 20:09:39.239971 18563 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac4a0 after lost signal to thread 18439
I20250622 20:09:40.380148 18880 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:09:40.380553 18880 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } }
I20250622 20:09:40.381976 18880 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107), b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463)
I20250622 20:09:40.382905 18522 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "4feef58d9e424d2899e5f8871a521526" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d5052250ec6041bfb6a01fe4ba815b43" is_pre_election: true
W20250622 20:09:40.387076 18590 leader_election.cc:336] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b27eab7a62c04658a0dcde9364054f20 (127.17.164.129:40463): Network error: Client connection negotiation failed: client connection to 127.17.164.129:40463: connect: Connection refused (error 111)
I20250622 20:09:40.387403 18590 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526; no voters: b27eab7a62c04658a0dcde9364054f20, d5052250ec6041bfb6a01fe4ba815b43
I20250622 20:09:40.387930 18880 raft_consensus.cc:2747] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250622 20:09:40.532326 18884 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250622 20:09:40.533378 18884 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true
I20250622 20:09:40.535655 18884 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397)
I20250622 20:09:40.545795 18656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "d5052250ec6041bfb6a01fe4ba815b43" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4feef58d9e424d2899e5f8871a521526" is_pre_election: true
I20250622 20:09:40.546263 18656 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5052250ec6041bfb6a01fe4ba815b43 in term 3.
I20250622 20:09:40.547206 18456 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526, d5052250ec6041bfb6a01fe4ba815b43; no voters:
I20250622 20:09:40.547897 18884 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250622 20:09:40.548188 18884 raft_consensus.cc:491] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250622 20:09:40.548450 18884 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:09:40.553651 18884 raft_consensus.cc:513] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true
I20250622 20:09:40.554667 18884 leader_election.cc:290] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [CANDIDATE]: Term 4 election: Requested vote from peers 4feef58d9e424d2899e5f8871a521526 (127.17.164.132:34397)
I20250622 20:09:40.555698 18656 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a66e023fcc594ca8a3267279f1ccba37" candidate_uuid: "d5052250ec6041bfb6a01fe4ba815b43" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4feef58d9e424d2899e5f8871a521526"
I20250622 20:09:40.556090 18656 raft_consensus.cc:3058] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:09:40.560127 18656 raft_consensus.cc:2466] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5052250ec6041bfb6a01fe4ba815b43 in term 4.
I20250622 20:09:40.561089 18456 leader_election.cc:304] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4feef58d9e424d2899e5f8871a521526, d5052250ec6041bfb6a01fe4ba815b43; no voters:
I20250622 20:09:40.561731 18884 raft_consensus.cc:2802] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 20:09:40.562722 18884 raft_consensus.cc:695] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 LEADER]: Becoming Leader. State: Replica: d5052250ec6041bfb6a01fe4ba815b43, State: Running, Role: LEADER
I20250622 20:09:40.563845 18884 consensus_queue.cc:237] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true
I20250622 20:09:40.573251 18787 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 reported cstate change: term changed from 2 to 4, leader changed from 4feef58d9e424d2899e5f8871a521526 (127.17.164.132) to d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131), now has a pending config: VOTER 4feef58d9e424d2899e5f8871a521526 (127.17.164.132), VOTER d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131). New cstate: current_term: 4 leader_uuid: "d5052250ec6041bfb6a01fe4ba815b43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b27eab7a62c04658a0dcde9364054f20" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 40463 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true }
I20250622 20:09:41.056176 18656 raft_consensus.cc:1273] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Refusing update from remote peer d5052250ec6041bfb6a01fe4ba815b43: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250622 20:09:41.057375 18884 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250622 20:09:41.065095 18895 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b27eab7a62c04658a0dcde9364054f20 (127.17.164.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true }
I20250622 20:09:41.066361 18656 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b27eab7a62c04658a0dcde9364054f20 (127.17.164.129) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } unsafe_config_change: true }
I20250622 20:09:41.074759 18787 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 reported cstate change: config changed from index -1 to 3, VOTER b27eab7a62c04658a0dcde9364054f20 (127.17.164.129) evicted, no longer has a pending config: VOTER 4feef58d9e424d2899e5f8871a521526 (127.17.164.132), VOTER d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131). New cstate: current_term: 4 leader_uuid: "d5052250ec6041bfb6a01fe4ba815b43" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250622 20:09:41.081355 18787 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet a66e023fcc594ca8a3267279f1ccba37 on TS b27eab7a62c04658a0dcde9364054f20: Not found: failed to reset TS proxy: Could not find TS for UUID b27eab7a62c04658a0dcde9364054f20
I20250622 20:09:41.096145 18522 consensus_queue.cc:237] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true
I20250622 20:09:41.100572 18656 raft_consensus.cc:1273] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Refusing update from remote peer d5052250ec6041bfb6a01fe4ba815b43: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250622 20:09:41.101938 18895 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250622 20:09:41.106878 18884 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true }
I20250622 20:09:41.108464 18656 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true }
W20250622 20:09:41.112149 18455 consensus_peers.cc:487] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 -> Peer 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130:35793): Couldn't send request to peer 1baa44b58f1546ef8bb0fd5072896b8b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a66e023fcc594ca8a3267279f1ccba37. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:09:41.114364 18773 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a66e023fcc594ca8a3267279f1ccba37 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 20:09:41.118091 18787 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 reported cstate change: config changed from index 3 to 5, NON_VOTER 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) added. New cstate: current_term: 4 leader_uuid: "d5052250ec6041bfb6a01fe4ba815b43" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true }
W20250622 20:09:41.134277 18773 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a66e023fcc594ca8a3267279f1ccba37 on TS b27eab7a62c04658a0dcde9364054f20 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b27eab7a62c04658a0dcde9364054f20
I20250622 20:09:41.670768 18904 ts_tablet_manager.cc:927] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Initiating tablet copy from peer d5052250ec6041bfb6a01fe4ba815b43 (127.17.164.131:38107)
I20250622 20:09:41.672950 18904 tablet_copy_client.cc:323] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: tablet copy: Beginning tablet copy session from remote peer at address 127.17.164.131:38107
I20250622 20:09:41.684317 18542 tablet_copy_service.cc:140] P d5052250ec6041bfb6a01fe4ba815b43: Received BeginTabletCopySession request for tablet a66e023fcc594ca8a3267279f1ccba37 from peer 1baa44b58f1546ef8bb0fd5072896b8b ({username='slave'} at 127.17.164.130:55611)
I20250622 20:09:41.684809 18542 tablet_copy_service.cc:161] P d5052250ec6041bfb6a01fe4ba815b43: Beginning new tablet copy session on tablet a66e023fcc594ca8a3267279f1ccba37 from peer 1baa44b58f1546ef8bb0fd5072896b8b at {username='slave'} at 127.17.164.130:55611: session id = 1baa44b58f1546ef8bb0fd5072896b8b-a66e023fcc594ca8a3267279f1ccba37
I20250622 20:09:41.689301 18542 tablet_copy_source_session.cc:215] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 20:09:41.693827 18904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66e023fcc594ca8a3267279f1ccba37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:41.711066 18904 tablet_copy_client.cc:806] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: tablet copy: Starting download of 0 data blocks...
I20250622 20:09:41.711578 18904 tablet_copy_client.cc:670] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: tablet copy: Starting download of 1 WAL segments...
I20250622 20:09:41.714974 18904 tablet_copy_client.cc:538] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 20:09:41.720258 18904 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Bootstrap starting.
I20250622 20:09:41.731084 18904 log.cc:826] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:41.741246 18904 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:09:41.741905 18904 tablet_bootstrap.cc:492] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Bootstrap complete.
I20250622 20:09:41.742424 18904 ts_tablet_manager.cc:1397] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.004s
I20250622 20:09:41.758682 18904 raft_consensus.cc:357] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true
I20250622 20:09:41.759552 18904 raft_consensus.cc:738] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 1baa44b58f1546ef8bb0fd5072896b8b, State: Initialized, Role: LEARNER
I20250622 20:09:41.760147 18904 consensus_queue.cc:260] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: true } } unsafe_config_change: true
I20250622 20:09:41.763758 18904 ts_tablet_manager.cc:1428] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b: Time spent starting tablet: real 0.021s user 0.021s sys 0.000s
I20250622 20:09:41.765317 18542 tablet_copy_service.cc:342] P d5052250ec6041bfb6a01fe4ba815b43: Request end of tablet copy session 1baa44b58f1546ef8bb0fd5072896b8b-a66e023fcc594ca8a3267279f1ccba37 received from {username='slave'} at 127.17.164.130:55611
I20250622 20:09:41.765661 18542 tablet_copy_service.cc:434] P d5052250ec6041bfb6a01fe4ba815b43: ending tablet copy session 1baa44b58f1546ef8bb0fd5072896b8b-a66e023fcc594ca8a3267279f1ccba37 on tablet a66e023fcc594ca8a3267279f1ccba37 with peer 1baa44b58f1546ef8bb0fd5072896b8b
I20250622 20:09:42.112241 18389 raft_consensus.cc:1215] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250622 20:09:42.300329 18773 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a66e023fcc594ca8a3267279f1ccba37 on TS b27eab7a62c04658a0dcde9364054f20 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b27eab7a62c04658a0dcde9364054f20
I20250622 20:09:42.625515 18910 raft_consensus.cc:1062] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43: attempting to promote NON_VOTER 1baa44b58f1546ef8bb0fd5072896b8b to VOTER
I20250622 20:09:42.626991 18910 consensus_queue.cc:237] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false } } unsafe_config_change: true
I20250622 20:09:42.631251 18389 raft_consensus.cc:1273] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [term 4 LEARNER]: Refusing update from remote peer d5052250ec6041bfb6a01fe4ba815b43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 20:09:42.631865 18656 raft_consensus.cc:1273] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Refusing update from remote peer d5052250ec6041bfb6a01fe4ba815b43: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250622 20:09:42.632498 18910 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 20:09:42.633500 18913 consensus_queue.cc:1035] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250622 20:09:42.640223 18910 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false } } unsafe_config_change: true }
I20250622 20:09:42.642212 18389 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P 1baa44b58f1546ef8bb0fd5072896b8b [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false } } unsafe_config_change: true }
I20250622 20:09:42.645359 18656 raft_consensus.cc:2953] T a66e023fcc594ca8a3267279f1ccba37 P 4feef58d9e424d2899e5f8871a521526 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false } } unsafe_config_change: true }
I20250622 20:09:42.651235 18787 catalog_manager.cc:5582] T a66e023fcc594ca8a3267279f1ccba37 P d5052250ec6041bfb6a01fe4ba815b43 reported cstate change: config changed from index 5 to 6, 1baa44b58f1546ef8bb0fd5072896b8b (127.17.164.130) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d5052250ec6041bfb6a01fe4ba815b43" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4feef58d9e424d2899e5f8871a521526" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 34397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35793 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250622 20:09:42.704131 18066 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250622 20:09:42.721350 18066 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "a66e023fcc594ca8a3267279f1ccba37"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "4feef58d9e424d2899e5f8871a521526"
rpc_addresses {
host: "127.17.164.132"
port: 34397
}
}
ts_infos {
permanent_uuid: "d5052250ec6041bfb6a01fe4ba815b43"
rpc_addresses {
host: "127.17.164.131"
port: 38107
}
}
ts_infos {
permanent_uuid: "1baa44b58f1546ef8bb0fd5072896b8b"
rpc_addresses {
host: "127.17.164.130"
port: 35793
}
}
I20250622 20:09:42.723677 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18305
I20250622 20:09:42.746742 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18438
I20250622 20:09:42.773133 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18571
I20250622 20:09:42.797935 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18739
2025-06-22T20:09:42Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19490 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250622 20:09:42.854298 18066 test_util.cc:276] Using random seed: -1392028029
I20250622 20:09:42.859853 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:09:42.860005 18066 ts_itest-base.cc:116] --------------
I20250622 20:09:42.860170 18066 ts_itest-base.cc:117] 3 tablet servers
I20250622 20:09:42.860302 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:09:42.860455 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:09:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:09:42Z Disabled control of system clock
I20250622 20:09:42.893908 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33643
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:33643
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250622 20:09:43.177881 18933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:43.178408 18933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:43.178843 18933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:43.212340 18933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:09:43.212671 18933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:43.212954 18933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:09:43.213179 18933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:09:43.246914 18933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:33643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:43.248152 18933 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:43.249739 18933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:43.263046 18939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:43.263067 18940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:43.265712 18933 server_base.cc:1048] running on GCE node
W20250622 20:09:43.265324 18942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:44.400893 18933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:44.403486 18933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:44.404825 18933 hybrid_clock.cc:648] HybridClock initialized: now 1750622984404799 us; error 43 us; skew 500 ppm
I20250622 20:09:44.405613 18933 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:44.411937 18933 webserver.cc:469] Webserver started at http://127.17.164.190:40861/ using document root <none> and password file <none>
I20250622 20:09:44.412884 18933 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:44.413081 18933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:44.413571 18933 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:44.417688 18933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fa32749bb6ad47ecbf05136ebb46b70d"
format_stamp: "Formatted at 2025-06-22 20:09:44 on dist-test-slave-9mkf"
I20250622 20:09:44.418717 18933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fa32749bb6ad47ecbf05136ebb46b70d"
format_stamp: "Formatted at 2025-06-22 20:09:44 on dist-test-slave-9mkf"
I20250622 20:09:44.425376 18933 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250622 20:09:44.431059 18950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:44.431996 18933 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 20:09:44.432289 18933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fa32749bb6ad47ecbf05136ebb46b70d"
format_stamp: "Formatted at 2025-06-22 20:09:44 on dist-test-slave-9mkf"
I20250622 20:09:44.432616 18933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:44.489970 18933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:44.491315 18933 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:44.491760 18933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:44.558266 18933 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:33643
I20250622 20:09:44.558337 19001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:33643 every 8 connection(s)
I20250622 20:09:44.561025 18933 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:09:44.565976 19002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:44.567853 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 18933
I20250622 20:09:44.568372 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:09:44.587952 19002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d: Bootstrap starting.
I20250622 20:09:44.595877 19002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:44.598074 19002 log.cc:826] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:44.603302 19002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d: No bootstrap required, opened a new log
I20250622 20:09:44.626469 19002 raft_consensus.cc:357] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } }
I20250622 20:09:44.627132 19002 raft_consensus.cc:383] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:44.627362 19002 raft_consensus.cc:738] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa32749bb6ad47ecbf05136ebb46b70d, State: Initialized, Role: FOLLOWER
I20250622 20:09:44.628062 19002 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } }
I20250622 20:09:44.628533 19002 raft_consensus.cc:397] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:09:44.628875 19002 raft_consensus.cc:491] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:09:44.629271 19002 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:44.635124 19002 raft_consensus.cc:513] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } }
I20250622 20:09:44.635756 19002 leader_election.cc:304] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa32749bb6ad47ecbf05136ebb46b70d; no voters:
I20250622 20:09:44.637331 19002 leader_election.cc:290] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:09:44.638041 19007 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:09:44.640089 19007 raft_consensus.cc:695] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [term 1 LEADER]: Becoming Leader. State: Replica: fa32749bb6ad47ecbf05136ebb46b70d, State: Running, Role: LEADER
I20250622 20:09:44.640987 19007 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } }
I20250622 20:09:44.642526 19002 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:09:44.652226 19009 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } } }
I20250622 20:09:44.653061 19009 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:44.652241 19008 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa32749bb6ad47ecbf05136ebb46b70d. Latest consensus state: current_term: 1 leader_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa32749bb6ad47ecbf05136ebb46b70d" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33643 } } }
I20250622 20:09:44.653863 19008 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:44.659853 19017 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:09:44.672500 19017 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:09:44.687932 19017 catalog_manager.cc:1349] Generated new cluster ID: d606d4adffeb4a92b73c7f015a70c223
I20250622 20:09:44.688264 19017 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:09:44.707687 19017 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:09:44.709118 19017 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:09:44.720274 19017 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fa32749bb6ad47ecbf05136ebb46b70d: Generated new TSK 0
I20250622 20:09:44.721341 19017 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:09:44.747510 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--builtin_ntp_servers=127.17.164.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250622 20:09:45.039019 19026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 20:09:45.039654 19026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:45.039942 19026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:45.040400 19026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:45.071197 19026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:45.072048 19026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:09:45.106006 19026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:45.107398 19026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:45.109094 19026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:45.126268 19033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:45.126955 19032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:45.127095 19026 server_base.cc:1048] running on GCE node
W20250622 20:09:45.127188 19035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:46.258378 19026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:46.261025 19026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:46.262449 19026 hybrid_clock.cc:648] HybridClock initialized: now 1750622986262383 us; error 66 us; skew 500 ppm
I20250622 20:09:46.263458 19026 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:46.270592 19026 webserver.cc:469] Webserver started at http://127.17.164.129:36125/ using document root <none> and password file <none>
I20250622 20:09:46.271762 19026 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:46.272014 19026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:46.272542 19026 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:46.279872 19026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5ec57e97efef4897a3cba77b73ee9c81"
format_stamp: "Formatted at 2025-06-22 20:09:46 on dist-test-slave-9mkf"
I20250622 20:09:46.281463 19026 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5ec57e97efef4897a3cba77b73ee9c81"
format_stamp: "Formatted at 2025-06-22 20:09:46 on dist-test-slave-9mkf"
I20250622 20:09:46.290530 19026 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250622 20:09:46.297385 19042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:46.298583 19026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 20:09:46.298980 19026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5ec57e97efef4897a3cba77b73ee9c81"
format_stamp: "Formatted at 2025-06-22 20:09:46 on dist-test-slave-9mkf"
I20250622 20:09:46.299376 19026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:46.356539 19026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:46.357939 19026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:46.358292 19026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:46.360530 19026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:46.364358 19026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:46.364538 19026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:46.364769 19026 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:46.364920 19026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:46.489673 19026 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:33255
I20250622 20:09:46.489775 19154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:33255 every 8 connection(s)
I20250622 20:09:46.492094 19026 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:09:46.494443 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19026
I20250622 20:09:46.494952 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:09:46.500902 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--builtin_ntp_servers=127.17.164.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 20:09:46.511128 19155 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33643
I20250622 20:09:46.511628 19155 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:46.512940 19155 heartbeater.cc:507] Master 127.17.164.190:33643 requested a full tablet report, sending...
I20250622 20:09:46.515924 18967 ts_manager.cc:194] Registered new tserver with Master: 5ec57e97efef4897a3cba77b73ee9c81 (127.17.164.129:33255)
I20250622 20:09:46.518942 18967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:46987
W20250622 20:09:46.791949 19159 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 20:09:46.792732 19159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:46.793080 19159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:46.793579 19159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:46.823140 19159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:46.823889 19159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:09:46.858578 19159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:46.859838 19159 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:46.861490 19159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:46.877816 19165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:47.522882 19155 heartbeater.cc:499] Master 127.17.164.190:33643 was elected leader, sending a full tablet report...
W20250622 20:09:46.878144 19168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:46.878728 19166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:46.880273 19159 server_base.cc:1048] running on GCE node
I20250622 20:09:47.993199 19159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:47.995904 19159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:47.997308 19159 hybrid_clock.cc:648] HybridClock initialized: now 1750622987997266 us; error 61 us; skew 500 ppm
I20250622 20:09:47.998055 19159 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:48.009431 19159 webserver.cc:469] Webserver started at http://127.17.164.130:38669/ using document root <none> and password file <none>
I20250622 20:09:48.010356 19159 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:48.010556 19159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:48.011013 19159 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:48.015223 19159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "599d6048b2f8490a8df5df8824e2d7ad"
format_stamp: "Formatted at 2025-06-22 20:09:48 on dist-test-slave-9mkf"
I20250622 20:09:48.016191 19159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "599d6048b2f8490a8df5df8824e2d7ad"
format_stamp: "Formatted at 2025-06-22 20:09:48 on dist-test-slave-9mkf"
I20250622 20:09:48.022841 19159 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 20:09:48.028621 19175 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:48.029873 19159 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 20:09:48.030238 19159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "599d6048b2f8490a8df5df8824e2d7ad"
format_stamp: "Formatted at 2025-06-22 20:09:48 on dist-test-slave-9mkf"
I20250622 20:09:48.030678 19159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:48.097064 19159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:48.098373 19159 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:48.098734 19159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:48.101125 19159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:48.104979 19159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:48.105158 19159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:48.105350 19159 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:48.105497 19159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:48.231325 19159 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:33221
I20250622 20:09:48.231442 19287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:33221 every 8 connection(s)
I20250622 20:09:48.233943 19159 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:09:48.239171 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19159
I20250622 20:09:48.239744 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:09:48.245688 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--builtin_ntp_servers=127.17.164.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250622 20:09:48.254448 19288 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33643
I20250622 20:09:48.254880 19288 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:48.255863 19288 heartbeater.cc:507] Master 127.17.164.190:33643 requested a full tablet report, sending...
I20250622 20:09:48.257984 18967 ts_manager.cc:194] Registered new tserver with Master: 599d6048b2f8490a8df5df8824e2d7ad (127.17.164.130:33221)
I20250622 20:09:48.259136 18967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:46271
W20250622 20:09:48.531947 19292 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250622 20:09:48.532563 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:48.532840 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:48.533305 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:48.563220 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:48.564258 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:09:48.598002 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:48.599294 19292 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:48.601045 19292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:48.614849 19298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:49.261942 19288 heartbeater.cc:499] Master 127.17.164.190:33643 was elected leader, sending a full tablet report...
W20250622 20:09:48.615702 19299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:48.633724 19292 server_base.cc:1048] running on GCE node
W20250622 20:09:48.633723 19301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:49.755084 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:49.757396 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:49.758745 19292 hybrid_clock.cc:648] HybridClock initialized: now 1750622989758717 us; error 45 us; skew 500 ppm
I20250622 20:09:49.759533 19292 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:49.766191 19292 webserver.cc:469] Webserver started at http://127.17.164.131:45487/ using document root <none> and password file <none>
I20250622 20:09:49.767109 19292 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:49.767315 19292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:49.767784 19292 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:49.772156 19292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3dd12a46a9434ed49bbcdb1220337784"
format_stamp: "Formatted at 2025-06-22 20:09:49 on dist-test-slave-9mkf"
I20250622 20:09:49.773240 19292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3dd12a46a9434ed49bbcdb1220337784"
format_stamp: "Formatted at 2025-06-22 20:09:49 on dist-test-slave-9mkf"
I20250622 20:09:49.780113 19292 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250622 20:09:49.785512 19308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:49.786553 19292 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250622 20:09:49.786834 19292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3dd12a46a9434ed49bbcdb1220337784"
format_stamp: "Formatted at 2025-06-22 20:09:49 on dist-test-slave-9mkf"
I20250622 20:09:49.787178 19292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:49.865651 19292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:49.867015 19292 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:49.867441 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:49.869891 19292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:09:49.873922 19292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:09:49.874096 19292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:49.874270 19292 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:09:49.874393 19292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:50.002770 19292 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:43039
I20250622 20:09:50.002892 19420 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:43039 every 8 connection(s)
I20250622 20:09:50.005275 19292 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:09:50.015504 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19292
I20250622 20:09:50.015908 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:09:50.025627 19421 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33643
I20250622 20:09:50.026038 19421 heartbeater.cc:461] Registering TS with master...
I20250622 20:09:50.027163 19421 heartbeater.cc:507] Master 127.17.164.190:33643 requested a full tablet report, sending...
I20250622 20:09:50.029333 18967 ts_manager.cc:194] Registered new tserver with Master: 3dd12a46a9434ed49bbcdb1220337784 (127.17.164.131:43039)
I20250622 20:09:50.030663 18967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:58385
I20250622 20:09:50.035251 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:09:50.070261 18967 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41878:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 20:09:50.088783 18967 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:09:50.144938 19090 tablet_service.cc:1468] Processing CreateTablet for tablet 3701a7fae168411e8a2360489105cbce (DEFAULT_TABLE table=TestTable [id=046e190c81034d5e8a4a595cb723a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:50.146868 19090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a7fae168411e8a2360489105cbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:50.147811 19356 tablet_service.cc:1468] Processing CreateTablet for tablet 3701a7fae168411e8a2360489105cbce (DEFAULT_TABLE table=TestTable [id=046e190c81034d5e8a4a595cb723a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:50.148059 19223 tablet_service.cc:1468] Processing CreateTablet for tablet 3701a7fae168411e8a2360489105cbce (DEFAULT_TABLE table=TestTable [id=046e190c81034d5e8a4a595cb723a8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:09:50.149328 19356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a7fae168411e8a2360489105cbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:50.149791 19223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a7fae168411e8a2360489105cbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:50.170295 19440 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Bootstrap starting.
I20250622 20:09:50.176578 19440 tablet_bootstrap.cc:654] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:50.178233 19441 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Bootstrap starting.
I20250622 20:09:50.178889 19440 log.cc:826] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:50.181047 19442 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Bootstrap starting.
I20250622 20:09:50.185370 19440 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: No bootstrap required, opened a new log
I20250622 20:09:50.185819 19441 tablet_bootstrap.cc:654] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:50.185922 19440 ts_tablet_manager.cc:1397] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20250622 20:09:50.187014 19442 tablet_bootstrap.cc:654] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:50.188697 19441 log.cc:826] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:50.189287 19442 log.cc:826] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:50.194427 19441 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: No bootstrap required, opened a new log
I20250622 20:09:50.194801 19442 tablet_bootstrap.cc:492] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: No bootstrap required, opened a new log
I20250622 20:09:50.194962 19441 ts_tablet_manager.cc:1397] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250622 20:09:50.195302 19442 ts_tablet_manager.cc:1397] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20250622 20:09:50.206909 19440 raft_consensus.cc:357] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.207566 19440 raft_consensus.cc:738] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd12a46a9434ed49bbcdb1220337784, State: Initialized, Role: FOLLOWER
I20250622 20:09:50.208164 19440 consensus_queue.cc:260] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.211097 19421 heartbeater.cc:499] Master 127.17.164.190:33643 was elected leader, sending a full tablet report...
I20250622 20:09:50.214032 19440 ts_tablet_manager.cc:1428] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Time spent starting tablet: real 0.028s user 0.024s sys 0.000s
I20250622 20:09:50.221551 19441 raft_consensus.cc:357] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.221562 19442 raft_consensus.cc:357] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.222442 19442 raft_consensus.cc:738] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599d6048b2f8490a8df5df8824e2d7ad, State: Initialized, Role: FOLLOWER
I20250622 20:09:50.222589 19441 raft_consensus.cc:738] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec57e97efef4897a3cba77b73ee9c81, State: Initialized, Role: FOLLOWER
I20250622 20:09:50.223182 19442 consensus_queue.cc:260] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.223224 19441 consensus_queue.cc:260] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.227923 19442 ts_tablet_manager.cc:1428] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Time spent starting tablet: real 0.032s user 0.029s sys 0.004s
I20250622 20:09:50.230288 19441 ts_tablet_manager.cc:1428] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Time spent starting tablet: real 0.035s user 0.022s sys 0.012s
W20250622 20:09:50.240469 19289 tablet.cc:2378] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:09:50.242727 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:09:50.246376 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5ec57e97efef4897a3cba77b73ee9c81 to finish bootstrapping
W20250622 20:09:50.249917 19156 tablet.cc:2378] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:09:50.259848 19422 tablet.cc:2378] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:09:50.261308 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 599d6048b2f8490a8df5df8824e2d7ad to finish bootstrapping
I20250622 20:09:50.274236 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3dd12a46a9434ed49bbcdb1220337784 to finish bootstrapping
I20250622 20:09:50.316288 19110 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3701a7fae168411e8a2360489105cbce"
dest_uuid: "5ec57e97efef4897a3cba77b73ee9c81"
from {username='slave'} at 127.0.0.1:45790
I20250622 20:09:50.316926 19110 raft_consensus.cc:491] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 20:09:50.317274 19110 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:50.321614 19110 raft_consensus.cc:513] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.323758 19110 leader_election.cc:290] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [CANDIDATE]: Term 1 election: Requested vote from peers 3dd12a46a9434ed49bbcdb1220337784 (127.17.164.131:43039), 599d6048b2f8490a8df5df8824e2d7ad (127.17.164.130:33221)
I20250622 20:09:50.335244 19375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3701a7fae168411e8a2360489105cbce" candidate_uuid: "5ec57e97efef4897a3cba77b73ee9c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3dd12a46a9434ed49bbcdb1220337784"
I20250622 20:09:50.335493 18066 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250622 20:09:50.336006 19375 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:50.338024 19243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3701a7fae168411e8a2360489105cbce" candidate_uuid: "5ec57e97efef4897a3cba77b73ee9c81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "599d6048b2f8490a8df5df8824e2d7ad"
I20250622 20:09:50.338814 19243 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:50.342839 19375 raft_consensus.cc:2466] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec57e97efef4897a3cba77b73ee9c81 in term 1.
I20250622 20:09:50.344149 19044 leader_election.cc:304] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd12a46a9434ed49bbcdb1220337784, 5ec57e97efef4897a3cba77b73ee9c81; no voters:
I20250622 20:09:50.345175 19447 raft_consensus.cc:2802] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:09:50.345204 19243 raft_consensus.cc:2466] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ec57e97efef4897a3cba77b73ee9c81 in term 1.
I20250622 20:09:50.347368 19447 raft_consensus.cc:695] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 LEADER]: Becoming Leader. State: Replica: 5ec57e97efef4897a3cba77b73ee9c81, State: Running, Role: LEADER
I20250622 20:09:50.348186 19447 consensus_queue.cc:237] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:50.359432 18965 catalog_manager.cc:5582] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ec57e97efef4897a3cba77b73ee9c81 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "5ec57e97efef4897a3cba77b73ee9c81" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } health_report { overall_health: UNKNOWN } } }
I20250622 20:09:50.441536 18066 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 20:09:50.646811 18066 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250622 20:09:50.935289 19455 consensus_queue.cc:1035] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:09:50.955474 19455 consensus_queue.cc:1035] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:09:50.957506 18066 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 1.1
I20250622 20:09:52.976857 19110 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3701a7fae168411e8a2360489105cbce"
dest_uuid: "5ec57e97efef4897a3cba77b73ee9c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45804
I20250622 20:09:52.977329 19110 raft_consensus.cc:604] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 LEADER]: Received request to transfer leadership
I20250622 20:09:53.012084 19486 raft_consensus.cc:991] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81: : Instructing follower 599d6048b2f8490a8df5df8824e2d7ad to start an election
I20250622 20:09:53.012511 19475 raft_consensus.cc:1079] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 LEADER]: Signalling peer 599d6048b2f8490a8df5df8824e2d7ad to start an election
I20250622 20:09:53.013881 19242 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "3701a7fae168411e8a2360489105cbce"
dest_uuid: "599d6048b2f8490a8df5df8824e2d7ad"
from {username='slave'} at 127.17.164.129:32943
I20250622 20:09:53.014365 19242 raft_consensus.cc:491] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 20:09:53.014619 19242 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:53.019004 19242 raft_consensus.cc:513] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:53.033409 19242 leader_election.cc:290] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [CANDIDATE]: Term 2 election: Requested vote from peers 5ec57e97efef4897a3cba77b73ee9c81 (127.17.164.129:33255), 3dd12a46a9434ed49bbcdb1220337784 (127.17.164.131:43039)
I20250622 20:09:53.043766 19110 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3701a7fae168411e8a2360489105cbce" candidate_uuid: "599d6048b2f8490a8df5df8824e2d7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ec57e97efef4897a3cba77b73ee9c81"
I20250622 20:09:53.044355 19110 raft_consensus.cc:3053] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 20:09:53.044596 19110 raft_consensus.cc:738] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ec57e97efef4897a3cba77b73ee9c81, State: Running, Role: LEADER
I20250622 20:09:53.045176 19110 consensus_queue.cc:260] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:53.046128 19110 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:53.047387 19375 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3701a7fae168411e8a2360489105cbce" candidate_uuid: "599d6048b2f8490a8df5df8824e2d7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3dd12a46a9434ed49bbcdb1220337784"
I20250622 20:09:53.047811 19375 raft_consensus.cc:3058] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:09:53.050544 19110 raft_consensus.cc:2466] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599d6048b2f8490a8df5df8824e2d7ad in term 2.
I20250622 20:09:53.051622 19178 leader_election.cc:304] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 599d6048b2f8490a8df5df8824e2d7ad, 5ec57e97efef4897a3cba77b73ee9c81; no voters:
I20250622 20:09:53.051880 19375 raft_consensus.cc:2466] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 599d6048b2f8490a8df5df8824e2d7ad in term 2.
I20250622 20:09:53.053943 19490 raft_consensus.cc:2802] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:09:53.055174 19490 raft_consensus.cc:695] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [term 2 LEADER]: Becoming Leader. State: Replica: 599d6048b2f8490a8df5df8824e2d7ad, State: Running, Role: LEADER
I20250622 20:09:53.056668 19490 consensus_queue.cc:237] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } }
I20250622 20:09:53.063747 18967 catalog_manager.cc:5582] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad reported cstate change: term changed from 1 to 2, leader changed from 5ec57e97efef4897a3cba77b73ee9c81 (127.17.164.129) to 599d6048b2f8490a8df5df8824e2d7ad (127.17.164.130). New cstate: current_term: 2 leader_uuid: "599d6048b2f8490a8df5df8824e2d7ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "599d6048b2f8490a8df5df8824e2d7ad" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 33221 } health_report { overall_health: HEALTHY } } }
I20250622 20:09:53.492187 19375 raft_consensus.cc:1273] T 3701a7fae168411e8a2360489105cbce P 3dd12a46a9434ed49bbcdb1220337784 [term 2 FOLLOWER]: Refusing update from remote peer 599d6048b2f8490a8df5df8824e2d7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 20:09:53.493373 19490 consensus_queue.cc:1035] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd12a46a9434ed49bbcdb1220337784" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 20:09:53.508280 19110 raft_consensus.cc:1273] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 2 FOLLOWER]: Refusing update from remote peer 599d6048b2f8490a8df5df8824e2d7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250622 20:09:53.511344 19490 consensus_queue.cc:1035] T 3701a7fae168411e8a2360489105cbce P 599d6048b2f8490a8df5df8824e2d7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec57e97efef4897a3cba77b73ee9c81" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250622 20:09:55.700098 19110 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3701a7fae168411e8a2360489105cbce"
dest_uuid: "5ec57e97efef4897a3cba77b73ee9c81"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56798
I20250622 20:09:55.700685 19110 raft_consensus.cc:604] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 2 FOLLOWER]: Received request to transfer leadership
I20250622 20:09:55.701040 19110 raft_consensus.cc:612] T 3701a7fae168411e8a2360489105cbce P 5ec57e97efef4897a3cba77b73ee9c81 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250622 20:09:56.735358 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19026
I20250622 20:09:56.759322 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19159
I20250622 20:09:56.783838 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19292
I20250622 20:09:56.807525 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 18933
2025-06-22T20:09:56Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14005 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250622 20:09:56.860235 18066 test_util.cc:276] Using random seed: -1378022097
I20250622 20:09:56.864188 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:09:56.864362 18066 ts_itest-base.cc:116] --------------
I20250622 20:09:56.864513 18066 ts_itest-base.cc:117] 3 tablet servers
I20250622 20:09:56.864666 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:09:56.864861 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:09:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:09:56Z Disabled control of system clock
I20250622 20:09:56.898201 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:35767
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:35767 with env {}
W20250622 20:09:57.184736 19534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:57.185305 19534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:57.185710 19534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:57.217011 19534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:09:57.217397 19534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:57.217622 19534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:09:57.217816 19534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:09:57.251616 19534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:35767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:57.252943 19534 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:57.254586 19534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:57.270422 19543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:57.271443 19534 server_base.cc:1048] running on GCE node
W20250622 20:09:57.270435 19541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:57.270483 19540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:09:58.407469 19534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:09:58.409993 19534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:09:58.411321 19534 hybrid_clock.cc:648] HybridClock initialized: now 1750622998411302 us; error 39 us; skew 500 ppm
I20250622 20:09:58.412055 19534 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:09:58.418234 19534 webserver.cc:469] Webserver started at http://127.17.164.190:45969/ using document root <none> and password file <none>
I20250622 20:09:58.419102 19534 fs_manager.cc:362] Metadata directory not provided
I20250622 20:09:58.419289 19534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:09:58.419775 19534 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:09:58.423913 19534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a59dbabc640749f99d1d7c966a83e38b"
format_stamp: "Formatted at 2025-06-22 20:09:58 on dist-test-slave-9mkf"
I20250622 20:09:58.424973 19534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a59dbabc640749f99d1d7c966a83e38b"
format_stamp: "Formatted at 2025-06-22 20:09:58 on dist-test-slave-9mkf"
I20250622 20:09:58.431470 19534 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 20:09:58.436946 19550 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:09:58.437938 19534 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 20:09:58.438243 19534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a59dbabc640749f99d1d7c966a83e38b"
format_stamp: "Formatted at 2025-06-22 20:09:58 on dist-test-slave-9mkf"
I20250622 20:09:58.438545 19534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:09:58.501441 19534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:09:58.502815 19534 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:09:58.503226 19534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:09:58.570452 19534 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:35767
I20250622 20:09:58.570549 19601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:35767 every 8 connection(s)
I20250622 20:09:58.573000 19534 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:09:58.577873 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19534
I20250622 20:09:58.578235 19602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:09:58.578388 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:09:58.599385 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b: Bootstrap starting.
I20250622 20:09:58.604435 19602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b: Neither blocks nor log segments found. Creating new log.
I20250622 20:09:58.606403 19602 log.cc:826] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b: Log is configured to *not* fsync() on all Append() calls
I20250622 20:09:58.610581 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b: No bootstrap required, opened a new log
I20250622 20:09:58.627401 19602 raft_consensus.cc:357] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } }
I20250622 20:09:58.627978 19602 raft_consensus.cc:383] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:09:58.628156 19602 raft_consensus.cc:738] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59dbabc640749f99d1d7c966a83e38b, State: Initialized, Role: FOLLOWER
I20250622 20:09:58.628726 19602 consensus_queue.cc:260] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } }
I20250622 20:09:58.629154 19602 raft_consensus.cc:397] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:09:58.629355 19602 raft_consensus.cc:491] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:09:58.629604 19602 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:09:58.633410 19602 raft_consensus.cc:513] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } }
I20250622 20:09:58.634047 19602 leader_election.cc:304] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a59dbabc640749f99d1d7c966a83e38b; no voters:
I20250622 20:09:58.636035 19602 leader_election.cc:290] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:09:58.636637 19607 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:09:58.638798 19607 raft_consensus.cc:695] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [term 1 LEADER]: Becoming Leader. State: Replica: a59dbabc640749f99d1d7c966a83e38b, State: Running, Role: LEADER
I20250622 20:09:58.639458 19607 consensus_queue.cc:237] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } }
I20250622 20:09:58.640313 19602 sys_catalog.cc:564] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:09:58.648931 19609 sys_catalog.cc:455] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a59dbabc640749f99d1d7c966a83e38b. Latest consensus state: current_term: 1 leader_uuid: "a59dbabc640749f99d1d7c966a83e38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } } }
I20250622 20:09:58.649811 19609 sys_catalog.cc:458] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:58.650277 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a59dbabc640749f99d1d7c966a83e38b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a59dbabc640749f99d1d7c966a83e38b" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 35767 } } }
I20250622 20:09:58.651152 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b [sys.catalog]: This master's current role is: LEADER
I20250622 20:09:58.655268 19616 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:09:58.664942 19616 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:09:58.680632 19616 catalog_manager.cc:1349] Generated new cluster ID: 8cbe37149b9c461f8e2216b7a1ebc130
I20250622 20:09:58.680972 19616 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:09:58.707298 19616 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:09:58.709208 19616 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:09:58.725147 19616 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a59dbabc640749f99d1d7c966a83e38b: Generated new TSK 0
I20250622 20:09:58.725898 19616 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:09:58.733594 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--builtin_ntp_servers=127.17.164.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 20:09:59.030485 19626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:09:59.031037 19626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:09:59.031524 19626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:09:59.062983 19626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:09:59.063814 19626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:09:59.098944 19626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:09:59.100256 19626 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:09:59.101858 19626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:09:59.118477 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:59.122797 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:09:59.118963 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:00.593984 19634 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1474 milliseconds
I20250622 20:10:00.594087 19626 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:00.595319 19626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:00.597504 19626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:00.598860 19626 hybrid_clock.cc:648] HybridClock initialized: now 1750623000598830 us; error 41 us; skew 500 ppm
I20250622 20:10:00.599660 19626 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:00.609963 19626 webserver.cc:469] Webserver started at http://127.17.164.129:36701/ using document root <none> and password file <none>
I20250622 20:10:00.610872 19626 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:00.611088 19626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:00.611521 19626 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:00.615873 19626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eed78e4c82394f398a8a925cb67a9490"
format_stamp: "Formatted at 2025-06-22 20:10:00 on dist-test-slave-9mkf"
I20250622 20:10:00.616946 19626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eed78e4c82394f398a8a925cb67a9490"
format_stamp: "Formatted at 2025-06-22 20:10:00 on dist-test-slave-9mkf"
I20250622 20:10:00.623543 19626 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 20:10:00.628858 19642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:00.629757 19626 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:10:00.630033 19626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eed78e4c82394f398a8a925cb67a9490"
format_stamp: "Formatted at 2025-06-22 20:10:00 on dist-test-slave-9mkf"
I20250622 20:10:00.630319 19626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:00.677879 19626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:00.679214 19626 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:00.679601 19626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:00.681982 19626 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:00.685763 19626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:00.685956 19626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:00.686213 19626 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:00.686384 19626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:00.811295 19626 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:45881
I20250622 20:10:00.811398 19754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:45881 every 8 connection(s)
I20250622 20:10:00.813843 19626 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:10:00.820318 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19626
I20250622 20:10:00.820688 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:10:00.826522 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--builtin_ntp_servers=127.17.164.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:00.833993 19755 heartbeater.cc:344] Connected to a master server at 127.17.164.190:35767
I20250622 20:10:00.834389 19755 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:00.835345 19755 heartbeater.cc:507] Master 127.17.164.190:35767 requested a full tablet report, sending...
I20250622 20:10:00.837620 19567 ts_manager.cc:194] Registered new tserver with Master: eed78e4c82394f398a8a925cb67a9490 (127.17.164.129:45881)
I20250622 20:10:00.839510 19567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:43177
W20250622 20:10:01.116135 19759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:01.116618 19759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:01.117118 19759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:01.148113 19759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:01.149014 19759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:10:01.182394 19759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:01.183759 19759 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:01.185693 19759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:01.202198 19766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:01.843231 19755 heartbeater.cc:499] Master 127.17.164.190:35767 was elected leader, sending a full tablet report...
W20250622 20:10:01.202286 19768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:01.204550 19759 server_base.cc:1048] running on GCE node
W20250622 20:10:01.202396 19765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:02.343492 19759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:02.345803 19759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:02.347159 19759 hybrid_clock.cc:648] HybridClock initialized: now 1750623002347112 us; error 67 us; skew 500 ppm
I20250622 20:10:02.347959 19759 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:02.354395 19759 webserver.cc:469] Webserver started at http://127.17.164.130:41771/ using document root <none> and password file <none>
I20250622 20:10:02.355381 19759 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:02.355592 19759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:02.356118 19759 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:02.360309 19759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c202dbe1d3cd4e1599461dce2394994e"
format_stamp: "Formatted at 2025-06-22 20:10:02 on dist-test-slave-9mkf"
I20250622 20:10:02.361455 19759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c202dbe1d3cd4e1599461dce2394994e"
format_stamp: "Formatted at 2025-06-22 20:10:02 on dist-test-slave-9mkf"
I20250622 20:10:02.368660 19759 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250622 20:10:02.373942 19775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:02.374963 19759 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250622 20:10:02.375238 19759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c202dbe1d3cd4e1599461dce2394994e"
format_stamp: "Formatted at 2025-06-22 20:10:02 on dist-test-slave-9mkf"
I20250622 20:10:02.375543 19759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:02.425987 19759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:02.427407 19759 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:02.427838 19759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:02.430051 19759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:02.433977 19759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:02.434165 19759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:02.434430 19759 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:02.434577 19759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:02.561480 19759 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:37587
I20250622 20:10:02.561592 19887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:37587 every 8 connection(s)
I20250622 20:10:02.564134 19759 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:10:02.564872 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19759
I20250622 20:10:02.565362 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:10:02.572887 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--builtin_ntp_servers=127.17.164.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:02.595603 19888 heartbeater.cc:344] Connected to a master server at 127.17.164.190:35767
I20250622 20:10:02.596036 19888 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:02.597088 19888 heartbeater.cc:507] Master 127.17.164.190:35767 requested a full tablet report, sending...
I20250622 20:10:02.599051 19567 ts_manager.cc:194] Registered new tserver with Master: c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587)
I20250622 20:10:02.600185 19567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:39089
W20250622 20:10:02.862934 19892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:02.863432 19892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:02.863927 19892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:02.893975 19892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:02.894793 19892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:10:02.930163 19892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:35767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:02.931421 19892 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:02.933053 19892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:02.947785 19899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:03.603145 19888 heartbeater.cc:499] Master 127.17.164.190:35767 was elected leader, sending a full tablet report...
W20250622 20:10:02.947893 19898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:02.949541 19901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:04.105355 19900 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1155 milliseconds
I20250622 20:10:04.105475 19892 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:04.106659 19892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:04.108556 19892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:04.109913 19892 hybrid_clock.cc:648] HybridClock initialized: now 1750623004109889 us; error 55 us; skew 500 ppm
I20250622 20:10:04.110653 19892 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:04.117154 19892 webserver.cc:469] Webserver started at http://127.17.164.131:46001/ using document root <none> and password file <none>
I20250622 20:10:04.118050 19892 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:04.118232 19892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:04.118680 19892 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:04.122834 19892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bc8daff3ad934c949d421f43cd724897"
format_stamp: "Formatted at 2025-06-22 20:10:04 on dist-test-slave-9mkf"
I20250622 20:10:04.123910 19892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bc8daff3ad934c949d421f43cd724897"
format_stamp: "Formatted at 2025-06-22 20:10:04 on dist-test-slave-9mkf"
I20250622 20:10:04.130497 19892 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250622 20:10:04.135730 19909 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:04.136694 19892 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 20:10:04.137009 19892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bc8daff3ad934c949d421f43cd724897"
format_stamp: "Formatted at 2025-06-22 20:10:04 on dist-test-slave-9mkf"
I20250622 20:10:04.137300 19892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:04.182065 19892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:04.183378 19892 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:04.183804 19892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:04.186223 19892 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:04.190021 19892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:04.190207 19892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:04.190448 19892 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:04.190587 19892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:04.318596 19892 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:41273
I20250622 20:10:04.318698 20021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:41273 every 8 connection(s)
I20250622 20:10:04.321079 19892 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:10:04.331723 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 19892
I20250622 20:10:04.332415 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:10:04.342082 20022 heartbeater.cc:344] Connected to a master server at 127.17.164.190:35767
I20250622 20:10:04.342432 20022 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:04.343297 20022 heartbeater.cc:507] Master 127.17.164.190:35767 requested a full tablet report, sending...
I20250622 20:10:04.345163 19567 ts_manager.cc:194] Registered new tserver with Master: bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:04.346321 19567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:60875
I20250622 20:10:04.352092 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:04.383129 19567 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46704:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 20:10:04.400818 19567 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:10:04.446367 19690 tablet_service.cc:1468] Processing CreateTablet for tablet 26fd42b9274041639fd0f354d8ad0089 (DEFAULT_TABLE table=TestTable [id=3aab3cb23db34d908e2909d46dd0cfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:04.446945 19823 tablet_service.cc:1468] Processing CreateTablet for tablet 26fd42b9274041639fd0f354d8ad0089 (DEFAULT_TABLE table=TestTable [id=3aab3cb23db34d908e2909d46dd0cfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:04.448443 19690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26fd42b9274041639fd0f354d8ad0089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.448884 19823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26fd42b9274041639fd0f354d8ad0089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.455024 19957 tablet_service.cc:1468] Processing CreateTablet for tablet 26fd42b9274041639fd0f354d8ad0089 (DEFAULT_TABLE table=TestTable [id=3aab3cb23db34d908e2909d46dd0cfff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:04.456935 19957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26fd42b9274041639fd0f354d8ad0089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.472275 20041 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Bootstrap starting.
I20250622 20:10:04.474920 20042 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Bootstrap starting.
I20250622 20:10:04.480022 20041 tablet_bootstrap.cc:654] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.482437 20042 tablet_bootstrap.cc:654] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.482427 20041 log.cc:826] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:04.482916 20043 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Bootstrap starting.
I20250622 20:10:04.485080 20042 log.cc:826] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:04.488399 20041 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: No bootstrap required, opened a new log
I20250622 20:10:04.488867 20041 ts_tablet_manager.cc:1397] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20250622 20:10:04.489427 20043 tablet_bootstrap.cc:654] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.490240 20042 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: No bootstrap required, opened a new log
I20250622 20:10:04.490690 20042 ts_tablet_manager.cc:1397] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20250622 20:10:04.491142 20043 log.cc:826] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:04.496194 20043 tablet_bootstrap.cc:492] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: No bootstrap required, opened a new log
I20250622 20:10:04.496551 20043 ts_tablet_manager.cc:1397] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250622 20:10:04.513563 20043 raft_consensus.cc:357] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.514245 20043 raft_consensus.cc:383] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.514508 20043 raft_consensus.cc:738] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8daff3ad934c949d421f43cd724897, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.514524 20041 raft_consensus.cc:357] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.515357 20041 raft_consensus.cc:383] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.515205 20043 consensus_queue.cc:260] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.515647 20041 raft_consensus.cc:738] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c202dbe1d3cd4e1599461dce2394994e, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.515864 20042 raft_consensus.cc:357] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.516798 20042 raft_consensus.cc:383] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.516475 20041 consensus_queue.cc:260] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.517119 20042 raft_consensus.cc:738] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed78e4c82394f398a8a925cb67a9490, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.518343 20022 heartbeater.cc:499] Master 127.17.164.190:35767 was elected leader, sending a full tablet report...
I20250622 20:10:04.518150 20042 consensus_queue.cc:260] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.519229 20043 ts_tablet_manager.cc:1428] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
I20250622 20:10:04.522737 20042 ts_tablet_manager.cc:1428] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Time spent starting tablet: real 0.032s user 0.027s sys 0.003s
I20250622 20:10:04.525429 20041 ts_tablet_manager.cc:1428] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Time spent starting tablet: real 0.036s user 0.029s sys 0.005s
W20250622 20:10:04.574360 19889 tablet.cc:2378] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:10:04.575719 20023 tablet.cc:2378] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:10:04.579324 19756 tablet.cc:2378] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:10:04.759724 20049 raft_consensus.cc:491] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:04.760162 20049 raft_consensus.cc:513] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.762529 20049 leader_election.cc:290] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587), bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:04.773169 19977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26fd42b9274041639fd0f354d8ad0089" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8daff3ad934c949d421f43cd724897" is_pre_election: true
I20250622 20:10:04.773222 19843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26fd42b9274041639fd0f354d8ad0089" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e" is_pre_election: true
I20250622 20:10:04.773903 19977 raft_consensus.cc:2466] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 0.
I20250622 20:10:04.773921 19843 raft_consensus.cc:2466] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 0.
I20250622 20:10:04.774940 19645 leader_election.cc:304] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8daff3ad934c949d421f43cd724897, eed78e4c82394f398a8a925cb67a9490; no voters:
I20250622 20:10:04.775600 20049 raft_consensus.cc:2802] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:10:04.775916 20049 raft_consensus.cc:491] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:10:04.776166 20049 raft_consensus.cc:3058] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:04.780261 20049 raft_consensus.cc:513] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.781497 20049 leader_election.cc:290] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 election: Requested vote from peers c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587), bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:04.782145 19843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26fd42b9274041639fd0f354d8ad0089" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e"
I20250622 20:10:04.782397 19977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26fd42b9274041639fd0f354d8ad0089" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8daff3ad934c949d421f43cd724897"
I20250622 20:10:04.782524 19843 raft_consensus.cc:3058] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:04.782820 19977 raft_consensus.cc:3058] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:04.786761 19843 raft_consensus.cc:2466] T 26fd42b9274041639fd0f354d8ad0089 P c202dbe1d3cd4e1599461dce2394994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 1.
I20250622 20:10:04.786895 19977 raft_consensus.cc:2466] T 26fd42b9274041639fd0f354d8ad0089 P bc8daff3ad934c949d421f43cd724897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 1.
I20250622 20:10:04.787688 19644 leader_election.cc:304] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c202dbe1d3cd4e1599461dce2394994e, eed78e4c82394f398a8a925cb67a9490; no voters:
I20250622 20:10:04.788283 20049 raft_consensus.cc:2802] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:04.789726 20049 raft_consensus.cc:695] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [term 1 LEADER]: Becoming Leader. State: Replica: eed78e4c82394f398a8a925cb67a9490, State: Running, Role: LEADER
I20250622 20:10:04.790385 20049 consensus_queue.cc:237] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.801031 19567 catalog_manager.cc:5582] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed78e4c82394f398a8a925cb67a9490 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "eed78e4c82394f398a8a925cb67a9490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } health_report { overall_health: UNKNOWN } } }
I20250622 20:10:04.824466 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:04.827687 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eed78e4c82394f398a8a925cb67a9490 to finish bootstrapping
I20250622 20:10:04.840298 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c202dbe1d3cd4e1599461dce2394994e to finish bootstrapping
I20250622 20:10:04.850261 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bc8daff3ad934c949d421f43cd724897 to finish bootstrapping
I20250622 20:10:04.861907 19567 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46704:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250622 20:10:04.863402 19567 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:10:04.878765 19690 tablet_service.cc:1468] Processing CreateTablet for tablet dd3b21c10c9147279e4aebec89b52b78 (DEFAULT_TABLE table=TestAnotherTable [id=85e9b9f6365a416684027529e006391a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 20:10:04.879397 19823 tablet_service.cc:1468] Processing CreateTablet for tablet dd3b21c10c9147279e4aebec89b52b78 (DEFAULT_TABLE table=TestAnotherTable [id=85e9b9f6365a416684027529e006391a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 20:10:04.879835 19690 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3b21c10c9147279e4aebec89b52b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.880414 19823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3b21c10c9147279e4aebec89b52b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.880506 19957 tablet_service.cc:1468] Processing CreateTablet for tablet dd3b21c10c9147279e4aebec89b52b78 (DEFAULT_TABLE table=TestAnotherTable [id=85e9b9f6365a416684027529e006391a]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250622 20:10:04.881542 19957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd3b21c10c9147279e4aebec89b52b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:04.891280 20041 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e: Bootstrap starting.
I20250622 20:10:04.893347 20042 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490: Bootstrap starting.
I20250622 20:10:04.895102 20043 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897: Bootstrap starting.
I20250622 20:10:04.896342 20041 tablet_bootstrap.cc:654] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.899032 20042 tablet_bootstrap.cc:654] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.901829 20043 tablet_bootstrap.cc:654] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:04.904508 20042 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490: No bootstrap required, opened a new log
I20250622 20:10:04.904637 20041 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e: No bootstrap required, opened a new log
I20250622 20:10:04.904934 20042 ts_tablet_manager.cc:1397] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250622 20:10:04.905165 20041 ts_tablet_manager.cc:1397] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20250622 20:10:04.907649 20043 tablet_bootstrap.cc:492] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897: No bootstrap required, opened a new log
I20250622 20:10:04.907583 20042 raft_consensus.cc:357] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.907606 20041 raft_consensus.cc:357] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.908087 20043 ts_tablet_manager.cc:1397] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20250622 20:10:04.908248 20041 raft_consensus.cc:383] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.908286 20042 raft_consensus.cc:383] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.908593 20042 raft_consensus.cc:738] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed78e4c82394f398a8a925cb67a9490, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.908581 20041 raft_consensus.cc:738] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c202dbe1d3cd4e1599461dce2394994e, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.909220 20042 consensus_queue.cc:260] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.909348 20041 consensus_queue.cc:260] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.911851 20041 ts_tablet_manager.cc:1428] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250622 20:10:04.912384 20042 ts_tablet_manager.cc:1428] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250622 20:10:04.913041 20043 raft_consensus.cc:357] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.913677 20043 raft_consensus.cc:383] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:04.913910 20043 raft_consensus.cc:738] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8daff3ad934c949d421f43cd724897, State: Initialized, Role: FOLLOWER
I20250622 20:10:04.914443 20043 consensus_queue.cc:260] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:04.916277 20043 ts_tablet_manager.cc:1428] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250622 20:10:05.173085 20049 consensus_queue.cc:1035] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:10:05.185979 20049 consensus_queue.cc:1035] T 26fd42b9274041639fd0f354d8ad0089 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:10:05.213177 20047 raft_consensus.cc:491] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:05.213583 20047 raft_consensus.cc:513] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.215695 20047 leader_election.cc:290] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed78e4c82394f398a8a925cb67a9490 (127.17.164.129:45881), c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587)
I20250622 20:10:05.217906 20048 raft_consensus.cc:491] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:05.218463 20048 raft_consensus.cc:513] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.219868 20049 raft_consensus.cc:491] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:05.220149 20049 raft_consensus.cc:513] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.221683 20049 leader_election.cc:290] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587), bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:05.223416 19977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8daff3ad934c949d421f43cd724897" is_pre_election: true
I20250622 20:10:05.223973 19977 raft_consensus.cc:2466] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 0.
I20250622 20:10:05.225016 19645 leader_election.cc:304] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8daff3ad934c949d421f43cd724897, eed78e4c82394f398a8a925cb67a9490; no voters:
I20250622 20:10:05.226258 19843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e" is_pre_election: true
I20250622 20:10:05.226785 19843 raft_consensus.cc:2466] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 0.
I20250622 20:10:05.227620 20049 raft_consensus.cc:2802] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:10:05.227954 20049 raft_consensus.cc:491] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:10:05.228288 20049 raft_consensus.cc:3058] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:05.228878 20048 leader_election.cc:290] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eed78e4c82394f398a8a925cb67a9490 (127.17.164.129:45881), bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:05.231858 19843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "bc8daff3ad934c949d421f43cd724897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e" is_pre_election: true
I20250622 20:10:05.232450 19843 raft_consensus.cc:2466] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8daff3ad934c949d421f43cd724897 in term 0.
I20250622 20:10:05.233733 19911 leader_election.cc:304] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc8daff3ad934c949d421f43cd724897, c202dbe1d3cd4e1599461dce2394994e; no voters:
I20250622 20:10:05.234553 20047 raft_consensus.cc:2802] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:10:05.234313 19710 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "bc8daff3ad934c949d421f43cd724897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed78e4c82394f398a8a925cb67a9490" is_pre_election: true
I20250622 20:10:05.234863 20047 raft_consensus.cc:491] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:10:05.235155 20047 raft_consensus.cc:3058] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:05.235579 20049 raft_consensus.cc:513] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.237972 19710 raft_consensus.cc:2391] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc8daff3ad934c949d421f43cd724897 in current term 1: Already voted for candidate eed78e4c82394f398a8a925cb67a9490 in this term.
I20250622 20:10:05.241562 20047 raft_consensus.cc:513] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.243644 20047 leader_election.cc:290] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [CANDIDATE]: Term 1 election: Requested vote from peers eed78e4c82394f398a8a925cb67a9490 (127.17.164.129:45881), c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587)
I20250622 20:10:05.243824 19843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e"
I20250622 20:10:05.244398 19843 raft_consensus.cc:3058] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:05.244446 19842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "bc8daff3ad934c949d421f43cd724897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c202dbe1d3cd4e1599461dce2394994e"
I20250622 20:10:05.245616 19709 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "c202dbe1d3cd4e1599461dce2394994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed78e4c82394f398a8a925cb67a9490" is_pre_election: true
I20250622 20:10:05.246434 19709 raft_consensus.cc:2391] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c202dbe1d3cd4e1599461dce2394994e in current term 1: Already voted for candidate eed78e4c82394f398a8a925cb67a9490 in this term.
I20250622 20:10:05.246532 19710 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "bc8daff3ad934c949d421f43cd724897" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed78e4c82394f398a8a925cb67a9490"
I20250622 20:10:05.247373 19710 raft_consensus.cc:2391] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc8daff3ad934c949d421f43cd724897 in current term 1: Already voted for candidate eed78e4c82394f398a8a925cb67a9490 in this term.
I20250622 20:10:05.247687 20049 leader_election.cc:290] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 election: Requested vote from peers c202dbe1d3cd4e1599461dce2394994e (127.17.164.130:37587), bc8daff3ad934c949d421f43cd724897 (127.17.164.131:41273)
I20250622 20:10:05.248520 19977 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "eed78e4c82394f398a8a925cb67a9490" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8daff3ad934c949d421f43cd724897"
I20250622 20:10:05.248764 19976 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dd3b21c10c9147279e4aebec89b52b78" candidate_uuid: "c202dbe1d3cd4e1599461dce2394994e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8daff3ad934c949d421f43cd724897" is_pre_election: true
I20250622 20:10:05.249212 19977 raft_consensus.cc:2391] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eed78e4c82394f398a8a925cb67a9490 in current term 1: Already voted for candidate bc8daff3ad934c949d421f43cd724897 in this term.
I20250622 20:10:05.248821 19912 leader_election.cc:304] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc8daff3ad934c949d421f43cd724897; no voters: c202dbe1d3cd4e1599461dce2394994e, eed78e4c82394f398a8a925cb67a9490
I20250622 20:10:05.249934 20047 raft_consensus.cc:2747] T dd3b21c10c9147279e4aebec89b52b78 P bc8daff3ad934c949d421f43cd724897 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250622 20:10:05.250532 19778 leader_election.cc:304] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c202dbe1d3cd4e1599461dce2394994e; no voters: bc8daff3ad934c949d421f43cd724897, eed78e4c82394f398a8a925cb67a9490
I20250622 20:10:05.250666 19843 raft_consensus.cc:2466] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed78e4c82394f398a8a925cb67a9490 in term 1.
I20250622 20:10:05.251164 20048 raft_consensus.cc:2747] T dd3b21c10c9147279e4aebec89b52b78 P c202dbe1d3cd4e1599461dce2394994e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250622 20:10:05.251456 19644 leader_election.cc:304] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c202dbe1d3cd4e1599461dce2394994e, eed78e4c82394f398a8a925cb67a9490; no voters: bc8daff3ad934c949d421f43cd724897
I20250622 20:10:05.252074 20049 raft_consensus.cc:2802] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:05.252434 20049 raft_consensus.cc:695] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [term 1 LEADER]: Becoming Leader. State: Replica: eed78e4c82394f398a8a925cb67a9490, State: Running, Role: LEADER
I20250622 20:10:05.253077 20049 consensus_queue.cc:237] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } }
I20250622 20:10:05.259805 19567 catalog_manager.cc:5582] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed78e4c82394f398a8a925cb67a9490 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "eed78e4c82394f398a8a925cb67a9490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eed78e4c82394f398a8a925cb67a9490" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 45881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 } health_report { overall_health: UNKNOWN } } }
W20250622 20:10:05.657102 20069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:05.657629 20069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250622 20:10:05.680814 20049 consensus_queue.cc:1035] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c202dbe1d3cd4e1599461dce2394994e" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 20:10:05.692747 20069 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250622 20:10:05.698685 20049 consensus_queue.cc:1035] T dd3b21c10c9147279e4aebec89b52b78 P eed78e4c82394f398a8a925cb67a9490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8daff3ad934c949d421f43cd724897" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 20:10:06.959933 20069 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.219s user 0.391s sys 0.826s
W20250622 20:10:06.960240 20069 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.219s user 0.391s sys 0.826s
W20250622 20:10:08.318352 20091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:08.319063 20091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:08.363183 20091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 20:10:09.620759 20091 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.216s user 0.422s sys 0.783s
W20250622 20:10:09.621088 20091 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.217s user 0.422s sys 0.786s
W20250622 20:10:10.977229 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:10.977821 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:11.007695 20106 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 20:10:12.253253 20106 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.203s user 0.441s sys 0.749s
W20250622 20:10:12.253536 20106 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.204s user 0.441s sys 0.749s
W20250622 20:10:13.639110 20123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:13.639640 20123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:13.669960 20123 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 20:10:14.948256 20123 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.238s user 0.446s sys 0.788s
W20250622 20:10:14.948537 20123 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.238s user 0.446s sys 0.788s
I20250622 20:10:16.019371 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19626
I20250622 20:10:16.055122 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19759
I20250622 20:10:16.080803 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19892
I20250622 20:10:16.105294 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 19534
2025-06-22T20:10:16Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19300 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250622 20:10:16.161412 18066 test_util.cc:276] Using random seed: -1358720920
I20250622 20:10:16.165400 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:10:16.165560 18066 ts_itest-base.cc:116] --------------
I20250622 20:10:16.165728 18066 ts_itest-base.cc:117] 3 tablet servers
I20250622 20:10:16.165871 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:10:16.166030 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:10:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:10:16Z Disabled control of system clock
I20250622 20:10:16.199570 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:46883
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:46883
--superuser_acl=no-such-user with env {}
W20250622 20:10:16.482849 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:16.483364 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:16.483780 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:16.513366 20144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:10:16.513631 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:16.513886 20144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:10:16.514077 20144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:10:16.547577 20144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:46883
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:16.548887 20144 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:16.550316 20144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:16.564461 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:16.564518 20150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:16.564555 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:17.758019 20152 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1191 milliseconds
I20250622 20:10:17.758133 20144 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:17.759361 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:17.762384 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:17.763792 20144 hybrid_clock.cc:648] HybridClock initialized: now 1750623017763783 us; error 50 us; skew 500 ppm
I20250622 20:10:17.764564 20144 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:17.775236 20144 webserver.cc:469] Webserver started at http://127.17.164.190:45027/ using document root <none> and password file <none>
I20250622 20:10:17.776113 20144 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:17.776291 20144 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:17.776687 20144 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:17.782599 20144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0549047b40c34152a0a749861fc08aa3"
format_stamp: "Formatted at 2025-06-22 20:10:17 on dist-test-slave-9mkf"
I20250622 20:10:17.783639 20144 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0549047b40c34152a0a749861fc08aa3"
format_stamp: "Formatted at 2025-06-22 20:10:17 on dist-test-slave-9mkf"
I20250622 20:10:17.790463 20144 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250622 20:10:17.795941 20160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:17.796909 20144 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:10:17.797212 20144 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0549047b40c34152a0a749861fc08aa3"
format_stamp: "Formatted at 2025-06-22 20:10:17 on dist-test-slave-9mkf"
I20250622 20:10:17.797528 20144 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:17.861433 20144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:17.862970 20144 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:17.863396 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:17.932667 20144 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:46883
I20250622 20:10:17.932827 20211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:46883 every 8 connection(s)
I20250622 20:10:17.935544 20144 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:10:17.940546 20212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:17.943917 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20144
I20250622 20:10:17.944337 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:10:17.964577 20212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3: Bootstrap starting.
I20250622 20:10:17.970942 20212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:17.972553 20212 log.cc:826] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:17.976994 20212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3: No bootstrap required, opened a new log
I20250622 20:10:17.993705 20212 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } }
I20250622 20:10:17.994488 20212 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:17.994784 20212 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0549047b40c34152a0a749861fc08aa3, State: Initialized, Role: FOLLOWER
I20250622 20:10:17.995514 20212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } }
I20250622 20:10:17.995975 20212 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:17.996192 20212 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:17.996444 20212 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:18.000226 20212 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } }
I20250622 20:10:18.000885 20212 leader_election.cc:304] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0549047b40c34152a0a749861fc08aa3; no voters:
I20250622 20:10:18.002539 20212 leader_election.cc:290] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:10:18.003263 20217 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:18.005735 20217 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 0549047b40c34152a0a749861fc08aa3, State: Running, Role: LEADER
I20250622 20:10:18.006507 20217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } }
I20250622 20:10:18.007498 20212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:10:18.016883 20218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0549047b40c34152a0a749861fc08aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } } }
I20250622 20:10:18.017043 20219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0549047b40c34152a0a749861fc08aa3. Latest consensus state: current_term: 1 leader_uuid: "0549047b40c34152a0a749861fc08aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0549047b40c34152a0a749861fc08aa3" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 46883 } } }
I20250622 20:10:18.017915 20218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:18.017951 20219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3 [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:18.020916 20227 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:10:18.033957 20227 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:10:18.050815 20227 catalog_manager.cc:1349] Generated new cluster ID: 6216dd5c88184a29880f7707075e7f9c
I20250622 20:10:18.051138 20227 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:10:18.072402 20227 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:10:18.073865 20227 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:10:18.086254 20227 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0549047b40c34152a0a749861fc08aa3: Generated new TSK 0
I20250622 20:10:18.087075 20227 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:10:18.105399 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--builtin_ntp_servers=127.17.164.148:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 20:10:18.414381 20236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:18.414892 20236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:18.415390 20236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:18.446719 20236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:18.447595 20236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:10:18.483412 20236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:18.484722 20236 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:18.486347 20236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:18.504222 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:18.504989 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:18.505529 20236 server_base.cc:1048] running on GCE node
W20250622 20:10:18.505131 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:19.682551 20236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:19.685241 20236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:19.686673 20236 hybrid_clock.cc:648] HybridClock initialized: now 1750623019686623 us; error 70 us; skew 500 ppm
I20250622 20:10:19.687417 20236 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:19.694301 20236 webserver.cc:469] Webserver started at http://127.17.164.129:33865/ using document root <none> and password file <none>
I20250622 20:10:19.695230 20236 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:19.695421 20236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:19.695884 20236 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:19.700137 20236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2"
format_stamp: "Formatted at 2025-06-22 20:10:19 on dist-test-slave-9mkf"
I20250622 20:10:19.701188 20236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2"
format_stamp: "Formatted at 2025-06-22 20:10:19 on dist-test-slave-9mkf"
I20250622 20:10:19.708540 20236 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250622 20:10:19.714205 20252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:19.715274 20236 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250622 20:10:19.715543 20236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2"
format_stamp: "Formatted at 2025-06-22 20:10:19 on dist-test-slave-9mkf"
I20250622 20:10:19.715833 20236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:19.765635 20236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:19.767000 20236 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:19.767436 20236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:19.769986 20236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:19.773871 20236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:19.774065 20236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:19.774263 20236 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:19.774405 20236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:19.903976 20236 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:33727
I20250622 20:10:19.904067 20364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:33727 every 8 connection(s)
I20250622 20:10:19.906662 20236 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:10:19.913280 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20236
I20250622 20:10:19.913798 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:10:19.921361 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--builtin_ntp_servers=127.17.164.148:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:19.930168 20365 heartbeater.cc:344] Connected to a master server at 127.17.164.190:46883
I20250622 20:10:19.930745 20365 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:19.932195 20365 heartbeater.cc:507] Master 127.17.164.190:46883 requested a full tablet report, sending...
I20250622 20:10:19.935492 20177 ts_manager.cc:194] Registered new tserver with Master: bdd8fc74fa0449728b29e0b2d2dd18c2 (127.17.164.129:33727)
I20250622 20:10:19.938134 20177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:54279
W20250622 20:10:20.219540 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:20.220084 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:20.220571 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:20.251194 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:20.252002 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:10:20.285995 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:20.287312 20369 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:20.288954 20369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:20.305696 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:20.942682 20365 heartbeater.cc:499] Master 127.17.164.190:46883 was elected leader, sending a full tablet report...
W20250622 20:10:20.305720 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:20.306010 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:21.446183 20377 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:10:21.446300 20369 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:21.450474 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:21.452639 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:21.454056 20369 hybrid_clock.cc:648] HybridClock initialized: now 1750623021454021 us; error 54 us; skew 500 ppm
I20250622 20:10:21.454921 20369 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:21.461527 20369 webserver.cc:469] Webserver started at http://127.17.164.130:42941/ using document root <none> and password file <none>
I20250622 20:10:21.462438 20369 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:21.462630 20369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:21.463152 20369 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:21.467417 20369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4ddd3e3fb90644c58832b4b369ff04bb"
format_stamp: "Formatted at 2025-06-22 20:10:21 on dist-test-slave-9mkf"
I20250622 20:10:21.468456 20369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4ddd3e3fb90644c58832b4b369ff04bb"
format_stamp: "Formatted at 2025-06-22 20:10:21 on dist-test-slave-9mkf"
I20250622 20:10:21.475143 20369 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 20:10:21.480703 20385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:21.481725 20369 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:10:21.482064 20369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4ddd3e3fb90644c58832b4b369ff04bb"
format_stamp: "Formatted at 2025-06-22 20:10:21 on dist-test-slave-9mkf"
I20250622 20:10:21.482390 20369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:21.552577 20369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:21.554046 20369 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:21.554490 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:21.556977 20369 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:21.560801 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:21.561062 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:21.561292 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:21.561444 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:21.689436 20369 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:36239
I20250622 20:10:21.689529 20497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:36239 every 8 connection(s)
I20250622 20:10:21.691972 20369 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:10:21.701201 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20369
I20250622 20:10:21.701586 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:10:21.707938 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--builtin_ntp_servers=127.17.164.148:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:21.712802 20498 heartbeater.cc:344] Connected to a master server at 127.17.164.190:46883
I20250622 20:10:21.713229 20498 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:21.714214 20498 heartbeater.cc:507] Master 127.17.164.190:46883 requested a full tablet report, sending...
I20250622 20:10:21.716441 20177 ts_manager.cc:194] Registered new tserver with Master: 4ddd3e3fb90644c58832b4b369ff04bb (127.17.164.130:36239)
I20250622 20:10:21.717725 20177 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:55885
W20250622 20:10:22.003096 20502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:22.003587 20502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:22.004063 20502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:22.038769 20502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:22.039685 20502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:10:22.074412 20502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:46883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:22.075697 20502 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:22.077337 20502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:22.093801 20508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:22.720530 20498 heartbeater.cc:499] Master 127.17.164.190:46883 was elected leader, sending a full tablet report...
W20250622 20:10:22.093842 20509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:22.096035 20502 server_base.cc:1048] running on GCE node
W20250622 20:10:22.094888 20511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:23.227308 20502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:23.229959 20502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:23.231384 20502 hybrid_clock.cc:648] HybridClock initialized: now 1750623023231330 us; error 71 us; skew 500 ppm
I20250622 20:10:23.232153 20502 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:23.241968 20502 webserver.cc:469] Webserver started at http://127.17.164.131:41121/ using document root <none> and password file <none>
I20250622 20:10:23.242872 20502 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:23.243072 20502 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:23.243536 20502 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:23.247764 20502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "909a02db8e5c4e2888dfe37b91412c5b"
format_stamp: "Formatted at 2025-06-22 20:10:23 on dist-test-slave-9mkf"
I20250622 20:10:23.248878 20502 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "909a02db8e5c4e2888dfe37b91412c5b"
format_stamp: "Formatted at 2025-06-22 20:10:23 on dist-test-slave-9mkf"
I20250622 20:10:23.256033 20502 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20250622 20:10:23.261399 20518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:23.262490 20502 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250622 20:10:23.262756 20502 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "909a02db8e5c4e2888dfe37b91412c5b"
format_stamp: "Formatted at 2025-06-22 20:10:23 on dist-test-slave-9mkf"
I20250622 20:10:23.263043 20502 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:23.313194 20502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:23.314615 20502 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:23.315026 20502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:23.317512 20502 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:23.321630 20502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:23.321812 20502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:23.322048 20502 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:23.322206 20502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:23.450984 20502 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:44555
I20250622 20:10:23.451121 20630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:44555 every 8 connection(s)
I20250622 20:10:23.453461 20502 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:10:23.457388 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20502
I20250622 20:10:23.458175 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:10:23.476897 20632 heartbeater.cc:344] Connected to a master server at 127.17.164.190:46883
I20250622 20:10:23.477358 20632 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:23.478454 20632 heartbeater.cc:507] Master 127.17.164.190:46883 requested a full tablet report, sending...
I20250622 20:10:23.480825 20176 ts_manager.cc:194] Registered new tserver with Master: 909a02db8e5c4e2888dfe37b91412c5b (127.17.164.131:44555)
I20250622 20:10:23.482129 20176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:50329
I20250622 20:10:23.494979 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:23.528370 20176 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51366:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250622 20:10:23.546633 20176 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:10:23.598270 20300 tablet_service.cc:1468] Processing CreateTablet for tablet e41351a2cd77424d9293e9ba8b22d7b8 (DEFAULT_TABLE table=TestTable [id=59e6d74fbc6a47159371b63136eb7aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:23.598841 20433 tablet_service.cc:1468] Processing CreateTablet for tablet e41351a2cd77424d9293e9ba8b22d7b8 (DEFAULT_TABLE table=TestTable [id=59e6d74fbc6a47159371b63136eb7aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:23.600066 20300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41351a2cd77424d9293e9ba8b22d7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:23.600664 20433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41351a2cd77424d9293e9ba8b22d7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:23.602815 20566 tablet_service.cc:1468] Processing CreateTablet for tablet e41351a2cd77424d9293e9ba8b22d7b8 (DEFAULT_TABLE table=TestTable [id=59e6d74fbc6a47159371b63136eb7aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:23.604192 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41351a2cd77424d9293e9ba8b22d7b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:23.626291 20651 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Bootstrap starting.
I20250622 20:10:23.627338 20653 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Bootstrap starting.
I20250622 20:10:23.629617 20652 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Bootstrap starting.
I20250622 20:10:23.633888 20653 tablet_bootstrap.cc:654] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:23.635656 20653 log.cc:826] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:23.635807 20651 tablet_bootstrap.cc:654] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:23.637284 20652 tablet_bootstrap.cc:654] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:23.638177 20651 log.cc:826] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:23.639662 20652 log.cc:826] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:23.641005 20653 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: No bootstrap required, opened a new log
I20250622 20:10:23.641546 20653 ts_tablet_manager.cc:1397] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20250622 20:10:23.643821 20651 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: No bootstrap required, opened a new log
I20250622 20:10:23.644284 20651 ts_tablet_manager.cc:1397] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.013s
I20250622 20:10:23.645709 20652 tablet_bootstrap.cc:492] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: No bootstrap required, opened a new log
I20250622 20:10:23.646270 20652 ts_tablet_manager.cc:1397] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.006s
I20250622 20:10:23.661391 20651 raft_consensus.cc:357] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.662329 20651 raft_consensus.cc:383] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:23.662634 20651 raft_consensus.cc:738] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909a02db8e5c4e2888dfe37b91412c5b, State: Initialized, Role: FOLLOWER
I20250622 20:10:23.663780 20651 consensus_queue.cc:260] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.666853 20632 heartbeater.cc:499] Master 127.17.164.190:46883 was elected leader, sending a full tablet report...
I20250622 20:10:23.668143 20651 ts_tablet_manager.cc:1428] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Time spent starting tablet: real 0.024s user 0.020s sys 0.002s
I20250622 20:10:23.667606 20653 raft_consensus.cc:357] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.668483 20653 raft_consensus.cc:383] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:23.668857 20653 raft_consensus.cc:738] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd8fc74fa0449728b29e0b2d2dd18c2, State: Initialized, Role: FOLLOWER
I20250622 20:10:23.669713 20653 consensus_queue.cc:260] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.672120 20652 raft_consensus.cc:357] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.673048 20652 raft_consensus.cc:383] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:23.673384 20652 raft_consensus.cc:738] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddd3e3fb90644c58832b4b369ff04bb, State: Initialized, Role: FOLLOWER
I20250622 20:10:23.673810 20653 ts_tablet_manager.cc:1428] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Time spent starting tablet: real 0.032s user 0.027s sys 0.005s
I20250622 20:10:23.674367 20652 consensus_queue.cc:260] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.679432 20652 ts_tablet_manager.cc:1428] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Time spent starting tablet: real 0.033s user 0.023s sys 0.008s
W20250622 20:10:23.698518 20499 tablet.cc:2378] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:10:23.715271 20633 tablet.cc:2378] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:10:23.840416 20657 raft_consensus.cc:491] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:23.840938 20657 raft_consensus.cc:513] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.843252 20657 leader_election.cc:290] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdd8fc74fa0449728b29e0b2d2dd18c2 (127.17.164.129:33727), 4ddd3e3fb90644c58832b4b369ff04bb (127.17.164.130:36239)
I20250622 20:10:23.851538 20320 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41351a2cd77424d9293e9ba8b22d7b8" candidate_uuid: "909a02db8e5c4e2888dfe37b91412c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" is_pre_election: true
I20250622 20:10:23.852459 20320 raft_consensus.cc:2466] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909a02db8e5c4e2888dfe37b91412c5b in term 0.
I20250622 20:10:23.853718 20521 leader_election.cc:304] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909a02db8e5c4e2888dfe37b91412c5b, bdd8fc74fa0449728b29e0b2d2dd18c2; no voters:
I20250622 20:10:23.853724 20453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41351a2cd77424d9293e9ba8b22d7b8" candidate_uuid: "909a02db8e5c4e2888dfe37b91412c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" is_pre_election: true
I20250622 20:10:23.854360 20657 raft_consensus.cc:2802] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:10:23.854411 20453 raft_consensus.cc:2466] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 909a02db8e5c4e2888dfe37b91412c5b in term 0.
I20250622 20:10:23.854598 20657 raft_consensus.cc:491] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:10:23.854884 20657 raft_consensus.cc:3058] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:23.859095 20657 raft_consensus.cc:513] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.860530 20657 leader_election.cc:290] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [CANDIDATE]: Term 1 election: Requested vote from peers bdd8fc74fa0449728b29e0b2d2dd18c2 (127.17.164.129:33727), 4ddd3e3fb90644c58832b4b369ff04bb (127.17.164.130:36239)
I20250622 20:10:23.861233 20320 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41351a2cd77424d9293e9ba8b22d7b8" candidate_uuid: "909a02db8e5c4e2888dfe37b91412c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2"
I20250622 20:10:23.861716 20320 raft_consensus.cc:3058] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:23.861735 20453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e41351a2cd77424d9293e9ba8b22d7b8" candidate_uuid: "909a02db8e5c4e2888dfe37b91412c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddd3e3fb90644c58832b4b369ff04bb"
I20250622 20:10:23.862192 20453 raft_consensus.cc:3058] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:23.866145 20320 raft_consensus.cc:2466] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909a02db8e5c4e2888dfe37b91412c5b in term 1.
I20250622 20:10:23.866490 20453 raft_consensus.cc:2466] T e41351a2cd77424d9293e9ba8b22d7b8 P 4ddd3e3fb90644c58832b4b369ff04bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 909a02db8e5c4e2888dfe37b91412c5b in term 1.
I20250622 20:10:23.866945 20521 leader_election.cc:304] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909a02db8e5c4e2888dfe37b91412c5b, bdd8fc74fa0449728b29e0b2d2dd18c2; no voters:
I20250622 20:10:23.867624 20657 raft_consensus.cc:2802] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:23.869226 20657 raft_consensus.cc:695] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [term 1 LEADER]: Becoming Leader. State: Replica: 909a02db8e5c4e2888dfe37b91412c5b, State: Running, Role: LEADER
I20250622 20:10:23.869944 20657 consensus_queue.cc:237] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } }
I20250622 20:10:23.880604 20177 catalog_manager.cc:5582] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 909a02db8e5c4e2888dfe37b91412c5b (127.17.164.131). New cstate: current_term: 1 leader_uuid: "909a02db8e5c4e2888dfe37b91412c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "909a02db8e5c4e2888dfe37b91412c5b" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 44555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 } health_report { overall_health: UNKNOWN } } }
W20250622 20:10:23.915503 20366 tablet.cc:2378] T e41351a2cd77424d9293e9ba8b22d7b8 P bdd8fc74fa0449728b29e0b2d2dd18c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:10:23.963585 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:23.967314 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bdd8fc74fa0449728b29e0b2d2dd18c2 to finish bootstrapping
I20250622 20:10:23.982503 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4ddd3e3fb90644c58832b4b369ff04bb to finish bootstrapping
I20250622 20:10:23.993697 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 909a02db8e5c4e2888dfe37b91412c5b to finish bootstrapping
I20250622 20:10:24.374635 20657 consensus_queue.cc:1035] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ddd3e3fb90644c58832b4b369ff04bb" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 20:10:24.406208 20676 consensus_queue.cc:1035] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd8fc74fa0449728b29e0b2d2dd18c2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250622 20:10:25.605532 20176 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:55790
I20250622 20:10:26.634690 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20236
I20250622 20:10:26.658758 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20369
W20250622 20:10:26.671836 20521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.129:33727: connect: Connection refused (error 111)
W20250622 20:10:26.675467 20521 consensus_peers.cc:487] T e41351a2cd77424d9293e9ba8b22d7b8 P 909a02db8e5c4e2888dfe37b91412c5b -> Peer bdd8fc74fa0449728b29e0b2d2dd18c2 (127.17.164.129:33727): Couldn't send request to peer bdd8fc74fa0449728b29e0b2d2dd18c2. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.129:33727: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:26.683490 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20502
I20250622 20:10:26.709272 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20144
2025-06-22T20:10:26Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10596 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250622 20:10:26.758136 18066 test_util.cc:276] Using random seed: -1348124193
I20250622 20:10:26.762012 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:10:26.762154 18066 ts_itest-base.cc:116] --------------
I20250622 20:10:26.762318 18066 ts_itest-base.cc:117] 3 tablet servers
I20250622 20:10:26.762456 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:10:26.762595 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:10:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:10:26Z Disabled control of system clock
I20250622 20:10:26.795707 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:33417 with env {}
W20250622 20:10:27.085994 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:27.086563 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:27.087000 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:27.116796 20700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:10:27.117139 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:27.117398 20700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:10:27.117645 20700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:10:27.151445 20700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:33417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:27.152742 20700 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:27.154374 20700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:27.170094 20706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:27.170085 20709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:27.172189 20700 server_base.cc:1048] running on GCE node
W20250622 20:10:27.170456 20707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:28.311503 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:28.314124 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:28.315479 20700 hybrid_clock.cc:648] HybridClock initialized: now 1750623028315444 us; error 55 us; skew 500 ppm
I20250622 20:10:28.316289 20700 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:28.325433 20700 webserver.cc:469] Webserver started at http://127.17.164.190:38957/ using document root <none> and password file <none>
I20250622 20:10:28.326351 20700 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:28.326547 20700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:28.326998 20700 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:28.332535 20700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:28.333652 20700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:28.340833 20700 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 20:10:28.346135 20716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:28.347074 20700 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250622 20:10:28.347376 20700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:28.347680 20700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:28.405845 20700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:28.407279 20700 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:28.407693 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:28.475868 20700 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:33417
I20250622 20:10:28.475942 20767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:33417 every 8 connection(s)
I20250622 20:10:28.478459 20700 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:10:28.482434 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20700
I20250622 20:10:28.482898 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:10:28.484304 20768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:28.506876 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap starting.
I20250622 20:10:28.511862 20768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:28.513469 20768 log.cc:826] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:28.517709 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: No bootstrap required, opened a new log
I20250622 20:10:28.534194 20768 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:28.534806 20768 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:28.535068 20768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Initialized, Role: FOLLOWER
I20250622 20:10:28.535832 20768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:28.536402 20768 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:28.536630 20768 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:28.536906 20768 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:28.540654 20768 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:28.541326 20768 leader_election.cc:304] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3594a994e5b9499d866dfc3d04585dfd; no voters:
I20250622 20:10:28.542855 20768 leader_election.cc:290] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:10:28.543568 20773 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:28.545686 20773 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 1 LEADER]: Becoming Leader. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Running, Role: LEADER
I20250622 20:10:28.546350 20773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:28.546689 20768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:10:28.557750 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3594a994e5b9499d866dfc3d04585dfd. Latest consensus state: current_term: 1 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:28.558445 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:28.560165 20774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:28.560837 20774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:28.561900 20783 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:10:28.573486 20783 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:10:28.589888 20783 catalog_manager.cc:1349] Generated new cluster ID: acd7998f65b7440fac8aee051820706b
I20250622 20:10:28.590140 20783 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:10:28.636431 20783 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:10:28.638505 20783 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:10:28.661185 20783 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Generated new TSK 0
I20250622 20:10:28.662307 20783 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:10:28.674888 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 20:10:28.972533 20792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:28.973071 20792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:28.973520 20792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:29.003556 20792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:29.005506 20792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:10:29.040174 20792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:29.041479 20792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:29.043035 20792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:29.060340 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:29.063477 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:29.064415 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:29.064603 20792 server_base.cc:1048] running on GCE node
I20250622 20:10:30.224254 20792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:30.226867 20792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:30.228319 20792 hybrid_clock.cc:648] HybridClock initialized: now 1750623030228271 us; error 50 us; skew 500 ppm
I20250622 20:10:30.229350 20792 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:30.236595 20792 webserver.cc:469] Webserver started at http://127.17.164.129:46875/ using document root <none> and password file <none>
I20250622 20:10:30.237725 20792 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:30.237969 20792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:30.238489 20792 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:30.244657 20792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2"
format_stamp: "Formatted at 2025-06-22 20:10:30 on dist-test-slave-9mkf"
I20250622 20:10:30.246098 20792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2"
format_stamp: "Formatted at 2025-06-22 20:10:30 on dist-test-slave-9mkf"
I20250622 20:10:30.254583 20792 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250622 20:10:30.261622 20808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:30.262673 20792 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250622 20:10:30.263000 20792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2"
format_stamp: "Formatted at 2025-06-22 20:10:30 on dist-test-slave-9mkf"
I20250622 20:10:30.263382 20792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:30.326357 20792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:30.327749 20792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:30.328164 20792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:30.330708 20792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:30.335275 20792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:30.335477 20792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:30.335711 20792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:30.335868 20792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:30.468957 20792 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:33203
I20250622 20:10:30.469106 20920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:33203 every 8 connection(s)
I20250622 20:10:30.472566 20792 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:10:30.477758 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20792
I20250622 20:10:30.478289 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:10:30.486341 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:30.510178 20921 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:30.510589 20921 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:30.511508 20921 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:30.513795 20733 ts_manager.cc:194] Registered new tserver with Master: d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203)
I20250622 20:10:30.515635 20733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:39665
W20250622 20:10:30.778144 20925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:30.778604 20925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:30.779069 20925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:30.808457 20925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:30.809310 20925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:10:30.842324 20925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:30.843592 20925 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:30.845173 20925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:30.861800 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:31.518702 20921 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
W20250622 20:10:30.861816 20934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:30.863317 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:30.864135 20925 server_base.cc:1048] running on GCE node
I20250622 20:10:31.972199 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:31.974314 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:31.975656 20925 hybrid_clock.cc:648] HybridClock initialized: now 1750623031975618 us; error 52 us; skew 500 ppm
I20250622 20:10:31.976404 20925 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:31.982257 20925 webserver.cc:469] Webserver started at http://127.17.164.130:37081/ using document root <none> and password file <none>
I20250622 20:10:31.983155 20925 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:31.983382 20925 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:31.983804 20925 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:31.987964 20925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "519cf2f27cab42d6a2f63949207804b7"
format_stamp: "Formatted at 2025-06-22 20:10:31 on dist-test-slave-9mkf"
I20250622 20:10:31.988963 20925 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "519cf2f27cab42d6a2f63949207804b7"
format_stamp: "Formatted at 2025-06-22 20:10:31 on dist-test-slave-9mkf"
I20250622 20:10:31.995307 20925 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 20:10:32.000222 20941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:32.001129 20925 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 20:10:32.001425 20925 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "519cf2f27cab42d6a2f63949207804b7"
format_stamp: "Formatted at 2025-06-22 20:10:31 on dist-test-slave-9mkf"
I20250622 20:10:32.001709 20925 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:32.056032 20925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:32.057439 20925 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:32.057921 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:32.060499 20925 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:32.064255 20925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:32.064460 20925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:32.064739 20925 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:32.064896 20925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250622 20:10:32.190021 20925 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:35337
I20250622 20:10:32.190140 21053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:35337 every 8 connection(s)
I20250622 20:10:32.192299 20925 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:10:32.197693 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 20925
I20250622 20:10:32.198071 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:10:32.203471 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:32.212227 21054 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:32.212635 21054 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:32.213587 21054 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:32.215621 20733 ts_manager.cc:194] Registered new tserver with Master: 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
I20250622 20:10:32.216853 20733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:34533
W20250622 20:10:32.487344 21058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:32.487835 21058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:32.488307 21058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:32.517244 21058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:32.518013 21058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:10:32.550627 21058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:32.551859 21058 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:32.553419 21058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:32.569701 21065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:33.219933 21054 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
W20250622 20:10:32.569816 21067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:32.571825 21058 server_base.cc:1048] running on GCE node
W20250622 20:10:32.569816 21064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:33.690089 21058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:33.692762 21058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:33.694185 21058 hybrid_clock.cc:648] HybridClock initialized: now 1750623033694135 us; error 76 us; skew 500 ppm
I20250622 20:10:33.694984 21058 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:33.705161 21058 webserver.cc:469] Webserver started at http://127.17.164.131:34049/ using document root <none> and password file <none>
I20250622 20:10:33.706065 21058 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:33.706245 21058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:33.706640 21058 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:10:33.710830 21058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9e51141a4c764f8db3e1f7a2395eaa97"
format_stamp: "Formatted at 2025-06-22 20:10:33 on dist-test-slave-9mkf"
I20250622 20:10:33.711794 21058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9e51141a4c764f8db3e1f7a2395eaa97"
format_stamp: "Formatted at 2025-06-22 20:10:33 on dist-test-slave-9mkf"
I20250622 20:10:33.718546 21058 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250622 20:10:33.723728 21075 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:33.724695 21058 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 20:10:33.724977 21058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e51141a4c764f8db3e1f7a2395eaa97"
format_stamp: "Formatted at 2025-06-22 20:10:33 on dist-test-slave-9mkf"
I20250622 20:10:33.725267 21058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:33.796125 21058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:33.797595 21058 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:33.797969 21058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:33.800275 21058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:33.804103 21058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:10:33.804281 21058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:33.804467 21058 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:10:33.804617 21058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:33.930799 21058 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:40057
I20250622 20:10:33.930919 21187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:40057 every 8 connection(s)
I20250622 20:10:33.933363 21058 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:10:33.941876 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 21058
I20250622 20:10:33.942394 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:10:33.953214 21188 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:33.953626 21188 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:33.954632 21188 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:33.956493 20733 ts_manager.cc:194] Registered new tserver with Master: 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:10:33.957692 20733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:37891
I20250622 20:10:33.961599 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:33.993176 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:33.993472 18066 test_util.cc:276] Using random seed: -1340888848
I20250622 20:10:34.035108 20733 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39498:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 20:10:34.073529 21123 tablet_service.cc:1468] Processing CreateTablet for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff (DEFAULT_TABLE table=TestTable [id=d8fb32b1f2e34472b20c7d05966aaa22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:34.075044 21123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:34.094110 21208 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap starting.
I20250622 20:10:34.099129 21208 tablet_bootstrap.cc:654] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:34.100690 21208 log.cc:826] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:34.105113 21208 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: No bootstrap required, opened a new log
I20250622 20:10:34.105497 21208 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250622 20:10:34.121287 21208 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:34.121784 21208 raft_consensus.cc:383] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:34.122022 21208 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Initialized, Role: FOLLOWER
I20250622 20:10:34.122745 21208 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:34.123208 21208 raft_consensus.cc:397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:34.123479 21208 raft_consensus.cc:491] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:34.123752 21208 raft_consensus.cc:3058] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:34.127902 21208 raft_consensus.cc:513] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:34.128548 21208 leader_election.cc:304] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e51141a4c764f8db3e1f7a2395eaa97; no voters:
I20250622 20:10:34.130151 21208 leader_election.cc:290] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:10:34.130514 21210 raft_consensus.cc:2802] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:34.132580 21210 raft_consensus.cc:695] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 LEADER]: Becoming Leader. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Running, Role: LEADER
I20250622 20:10:34.133464 21188 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:10:34.133529 21210 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:34.134449 21208 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent starting tablet: real 0.029s user 0.020s sys 0.008s
I20250622 20:10:34.146039 20732 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131). New cstate: current_term: 1 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } }
I20250622 20:10:34.375502 18066 test_util.cc:276] Using random seed: -1340506831
I20250622 20:10:34.395031 20731 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39502:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 20:10:34.421289 20856 tablet_service.cc:1468] Processing CreateTablet for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 (DEFAULT_TABLE table=TestTable1 [id=0c257710de3b410296083c3c61900a4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:34.422560 20856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0e9bf7b0d0491d8fdc4be911914da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:34.440439 21229 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap starting.
I20250622 20:10:34.445858 21229 tablet_bootstrap.cc:654] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:34.447510 21229 log.cc:826] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:34.451980 21229 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: No bootstrap required, opened a new log
I20250622 20:10:34.452381 21229 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250622 20:10:34.469579 21229 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:34.470084 21229 raft_consensus.cc:383] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:34.470326 21229 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Initialized, Role: FOLLOWER
I20250622 20:10:34.470916 21229 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:34.471393 21229 raft_consensus.cc:397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:34.471645 21229 raft_consensus.cc:491] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:34.471916 21229 raft_consensus.cc:3058] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:34.476994 21229 raft_consensus.cc:513] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:34.477766 21229 leader_election.cc:304] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85aa688d5bb48ceac0f2645bcfb7ad2; no voters:
I20250622 20:10:34.480748 21229 leader_election.cc:290] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:10:34.481297 21231 raft_consensus.cc:2802] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:34.493327 21231 raft_consensus.cc:695] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 LEADER]: Becoming Leader. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Running, Role: LEADER
I20250622 20:10:34.494233 21231 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:34.497315 21229 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent starting tablet: real 0.045s user 0.026s sys 0.020s
W20250622 20:10:34.499893 20922 tablet.cc:2378] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:10:34.509356 20731 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } }
I20250622 20:10:34.704006 18066 test_util.cc:276] Using random seed: -1340178324
I20250622 20:10:34.724149 20728 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39512:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 20:10:34.751279 20989 tablet_service.cc:1468] Processing CreateTablet for tablet bd5bfc94658b418db7da4d1d4a97c5d0 (DEFAULT_TABLE table=TestTable2 [id=810e070ace744ae1b962c5c441e20eb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:10:34.752619 20989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5bfc94658b418db7da4d1d4a97c5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:34.770167 21250 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:34.775879 21250 tablet_bootstrap.cc:654] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Neither blocks nor log segments found. Creating new log.
I20250622 20:10:34.777479 21250 log.cc:826] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:34.781380 21250 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: No bootstrap required, opened a new log
I20250622 20:10:34.781710 21250 ts_tablet_manager.cc:1397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250622 20:10:34.798450 21250 raft_consensus.cc:357] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:34.798911 21250 raft_consensus.cc:383] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:10:34.799090 21250 raft_consensus.cc:738] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: FOLLOWER
I20250622 20:10:34.799641 21250 consensus_queue.cc:260] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:34.800086 21250 raft_consensus.cc:397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:34.800303 21250 raft_consensus.cc:491] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:34.800537 21250 raft_consensus.cc:3058] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:10:34.804308 21250 raft_consensus.cc:513] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:34.804917 21250 leader_election.cc:304] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7; no voters:
I20250622 20:10:34.806578 21250 leader_election.cc:290] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:10:34.806916 21252 raft_consensus.cc:2802] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:10:34.810341 21250 ts_tablet_manager.cc:1428] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.028s user 0.018s sys 0.010s
I20250622 20:10:34.810357 21252 raft_consensus.cc:695] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 LEADER]: Becoming Leader. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Running, Role: LEADER
I20250622 20:10:34.811087 21252 consensus_queue.cc:237] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:34.821400 20728 catalog_manager.cc:5582] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130). New cstate: current_term: 1 leader_uuid: "519cf2f27cab42d6a2f63949207804b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } health_report { overall_health: HEALTHY } } }
I20250622 20:10:35.029541 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20700
W20250622 20:10:35.160755 21188 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
W20250622 20:10:35.530460 20921 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
W20250622 20:10:35.839437 21054 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
I20250622 20:10:39.605127 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20792
I20250622 20:10:39.627414 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 20925
I20250622 20:10:39.653108 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 21058
I20250622 20:10:39.678781 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--webserver_interface=127.17.164.190
--webserver_port=38957
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:33417 with env {}
W20250622 20:10:39.967180 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:39.967713 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:39.968096 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:39.997637 21328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:10:39.997897 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:39.998085 21328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:10:39.998260 21328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:10:40.031442 21328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:33417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=38957
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:40.033218 21328 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:40.035354 21328 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:40.050596 21334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:40.051352 21337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:40.051519 21335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:41.149317 21336 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:10:41.149369 21328 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:41.152758 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:41.155181 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:41.156499 21328 hybrid_clock.cc:648] HybridClock initialized: now 1750623041156476 us; error 64 us; skew 500 ppm
I20250622 20:10:41.157348 21328 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:41.167546 21328 webserver.cc:469] Webserver started at http://127.17.164.190:38957/ using document root <none> and password file <none>
I20250622 20:10:41.168406 21328 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:41.168594 21328 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:41.176187 21328 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250622 20:10:41.180593 21344 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:41.181640 21328 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250622 20:10:41.181941 21328 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:41.183714 21328 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:41.245415 21328 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:41.246810 21328 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:41.247223 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:41.316267 21328 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:33417
I20250622 20:10:41.316426 21395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:33417 every 8 connection(s)
I20250622 20:10:41.319038 21328 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:10:41.326890 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 21328
I20250622 20:10:41.328269 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:33203
--local_ip_for_outbound_sockets=127.17.164.129
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=46875
--webserver_interface=127.17.164.129
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:41.328367 21396 sys_catalog.cc:263] Verifying existing consensus state
I20250622 20:10:41.332656 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap starting.
I20250622 20:10:41.342156 21396 log.cc:826] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:41.388675 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=13 ignored=0} mutations{seen=11 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:41.389484 21396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap complete.
I20250622 20:10:41.408740 21396 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:41.410702 21396 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Initialized, Role: FOLLOWER
I20250622 20:10:41.411430 21396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:41.411873 21396 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:41.412156 21396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:41.412423 21396 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:10:41.417503 21396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:41.418074 21396 leader_election.cc:304] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3594a994e5b9499d866dfc3d04585dfd; no voters:
I20250622 20:10:41.420123 21396 leader_election.cc:290] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 20:10:41.420576 21400 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:10:41.423733 21400 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 LEADER]: Becoming Leader. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Running, Role: LEADER
I20250622 20:10:41.424609 21400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 2.19, Last appended by leader: 19, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:41.425082 21396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:10:41.435115 21401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:41.435274 21402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3594a994e5b9499d866dfc3d04585dfd. Latest consensus state: current_term: 3 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:41.435978 21402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:41.435978 21401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:41.439177 21407 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:10:41.451787 21407 catalog_manager.cc:671] Loaded metadata for table TestTable [id=118e917d48d34d569a80d76cd09ce814]
I20250622 20:10:41.453413 21407 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=810e070ace744ae1b962c5c441e20eb6]
I20250622 20:10:41.454941 21407 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=84c5c80c9eb545c9bcfd92175c478c37]
I20250622 20:10:41.462160 21407 tablet_loader.cc:96] loaded metadata for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 (table TestTable1 [id=84c5c80c9eb545c9bcfd92175c478c37])
I20250622 20:10:41.463403 21407 tablet_loader.cc:96] loaded metadata for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff (table TestTable [id=118e917d48d34d569a80d76cd09ce814])
I20250622 20:10:41.464663 21407 tablet_loader.cc:96] loaded metadata for tablet bd5bfc94658b418db7da4d1d4a97c5d0 (table TestTable2 [id=810e070ace744ae1b962c5c441e20eb6])
I20250622 20:10:41.466027 21407 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:10:41.470784 21407 catalog_manager.cc:1261] Loaded cluster ID: acd7998f65b7440fac8aee051820706b
I20250622 20:10:41.471067 21407 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:10:41.478693 21407 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:10:41.483438 21407 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Loaded TSK: 0
I20250622 20:10:41.484966 21407 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 20:10:41.646201 21398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:41.646704 21398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:41.647197 21398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:41.677599 21398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:41.678423 21398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:10:41.712056 21398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:33203
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=46875
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:41.713335 21398 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:41.714893 21398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:41.731426 21424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:41.732474 21426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:41.734478 21398 server_base.cc:1048] running on GCE node
W20250622 20:10:41.732940 21423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:42.864243 21398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:42.866979 21398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:42.868453 21398 hybrid_clock.cc:648] HybridClock initialized: now 1750623042868406 us; error 51 us; skew 500 ppm
I20250622 20:10:42.869539 21398 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:42.876816 21398 webserver.cc:469] Webserver started at http://127.17.164.129:46875/ using document root <none> and password file <none>
I20250622 20:10:42.877960 21398 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:42.878238 21398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:42.888236 21398 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 20:10:42.893498 21433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:42.894537 21398 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 20:10:42.894878 21398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2"
format_stamp: "Formatted at 2025-06-22 20:10:30 on dist-test-slave-9mkf"
I20250622 20:10:42.897482 21398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:42.970916 21398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:42.972765 21398 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:42.973333 21398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:42.976269 21398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:42.983242 21440 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:10:42.990116 21398 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:10:42.990370 21398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.001s
I20250622 20:10:42.990658 21398 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:10:42.997514 21398 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:10:42.997768 21398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20250622 20:10:42.997998 21440 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap starting.
I20250622 20:10:43.063038 21440 log.cc:826] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:43.173555 21398 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:33203
I20250622 20:10:43.173761 21547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:33203 every 8 connection(s)
I20250622 20:10:43.176990 21398 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:10:43.177963 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 21398
I20250622 20:10:43.179664 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:35337
--local_ip_for_outbound_sockets=127.17.164.130
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=37081
--webserver_interface=127.17.164.130
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:43.190750 21440 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:43.191780 21440 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap complete.
I20250622 20:10:43.193519 21440 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent bootstrapping tablet: real 0.196s user 0.145s sys 0.044s
I20250622 20:10:43.227689 21440 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:43.230474 21440 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Initialized, Role: FOLLOWER
I20250622 20:10:43.231465 21440 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:43.232138 21440 raft_consensus.cc:397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:43.232486 21440 raft_consensus.cc:491] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:43.232895 21440 raft_consensus.cc:3058] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:10:43.239145 21548 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:43.239557 21548 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:43.240614 21548 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:43.241390 21440 raft_consensus.cc:513] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:43.242026 21440 leader_election.cc:304] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d85aa688d5bb48ceac0f2645bcfb7ad2; no voters:
I20250622 20:10:43.253145 21361 ts_manager.cc:194] Registered new tserver with Master: d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203)
I20250622 20:10:43.261260 21440 leader_election.cc:290] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:10:43.262145 21553 raft_consensus.cc:2802] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:10:43.264470 21361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:53513
I20250622 20:10:43.274116 21553 raft_consensus.cc:695] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEADER]: Becoming Leader. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Running, Role: LEADER
I20250622 20:10:43.274186 21548 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:10:43.275038 21553 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } }
I20250622 20:10:43.276573 21440 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent starting tablet: real 0.083s user 0.033s sys 0.031s
I20250622 20:10:43.282639 21361 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: term changed from 0 to 2, leader changed from <none> to d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129), VOTER d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) added. New cstate: current_term: 2 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } }
W20250622 20:10:43.314642 21361 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with cas_config_opid_index -1: no extra replica candidate found for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 (table TestTable1 [id=84c5c80c9eb545c9bcfd92175c478c37]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250622 20:10:43.514529 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:43.514991 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:43.515444 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:43.546864 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:43.547634 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:10:43.581269 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:35337
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=37081
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:43.582461 21549 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:43.583976 21549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:43.599081 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:43.600194 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:43.600479 21569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:43.600745 21549 server_base.cc:1048] running on GCE node
I20250622 20:10:44.708974 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:44.711594 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:44.713011 21549 hybrid_clock.cc:648] HybridClock initialized: now 1750623044712976 us; error 44 us; skew 500 ppm
I20250622 20:10:44.713722 21549 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:44.720197 21549 webserver.cc:469] Webserver started at http://127.17.164.130:37081/ using document root <none> and password file <none>
I20250622 20:10:44.721179 21549 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:44.721375 21549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:44.728447 21549 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20250622 20:10:44.732445 21578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:44.733388 21549 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:10:44.733640 21549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "519cf2f27cab42d6a2f63949207804b7"
format_stamp: "Formatted at 2025-06-22 20:10:31 on dist-test-slave-9mkf"
I20250622 20:10:44.735411 21549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:44.787514 21549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:44.788839 21549 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:44.789191 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:44.791376 21549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:44.796392 21585 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:10:44.803661 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:10:44.803856 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250622 20:10:44.804083 21549 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:10:44.808248 21549 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:10:44.808405 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.001s sys 0.000s
I20250622 20:10:44.808830 21585 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:44.859241 21585 log.cc:826] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:44.938551 21585 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:44.939267 21585 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:10:44.940418 21585 ts_tablet_manager.cc:1397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.132s user 0.111s sys 0.015s
I20250622 20:10:44.955696 21585 raft_consensus.cc:357] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:44.958211 21585 raft_consensus.cc:738] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: FOLLOWER
I20250622 20:10:44.959064 21585 consensus_queue.cc:260] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:44.959507 21585 raft_consensus.cc:397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:44.959952 21585 raft_consensus.cc:491] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:44.960343 21585 raft_consensus.cc:3058] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:10:44.966152 21585 raft_consensus.cc:513] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:44.966825 21585 leader_election.cc:304] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7; no voters:
I20250622 20:10:44.969887 21585 leader_election.cc:290] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:10:44.970209 21689 raft_consensus.cc:2802] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:10:44.974289 21689 raft_consensus.cc:695] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEADER]: Becoming Leader. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Running, Role: LEADER
I20250622 20:10:44.975620 21689 consensus_queue.cc:237] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:10:44.978250 21549 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:35337
I20250622 20:10:44.978506 21695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:35337 every 8 connection(s)
I20250622 20:10:44.978688 21585 ts_tablet_manager.cc:1428] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.038s user 0.031s sys 0.008s
I20250622 20:10:44.981802 21549 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:10:44.986133 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 21549
I20250622 20:10:44.988246 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:40057
--local_ip_for_outbound_sockets=127.17.164.131
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=34049
--webserver_interface=127.17.164.131
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:44.999245 21698 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:44.999727 21698 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:45.000846 21698 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:45.004773 21361 ts_manager.cc:194] Registered new tserver with Master: 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
I20250622 20:10:45.006235 21361 catalog_manager.cc:5582] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "519cf2f27cab42d6a2f63949207804b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } health_report { overall_health: HEALTHY } } }
I20250622 20:10:45.018736 21361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:51221
I20250622 20:10:45.023111 21698 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
W20250622 20:10:45.287586 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:45.288082 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:45.288558 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:45.321043 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:45.322368 21702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:10:45.376843 21702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:40057
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=34049
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:45.378082 21702 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:45.379606 21702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:45.394526 21712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:45.628403 21503 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:45.635141 21722 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } } }
I20250622 20:10:45.647806 21346 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250622 20:10:45.652143 21361 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index -1 to 9, NON_VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) added. New cstate: current_term: 2 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250622 20:10:45.660099 21437 consensus_peers.cc:487] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 -> Peer 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337): Couldn't send request to peer 519cf2f27cab42d6a2f63949207804b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f0e9bf7b0d0491d8fdc4be911914da5. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:46.096308 21728 ts_tablet_manager.cc:927] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Initiating tablet copy from peer d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203)
I20250622 20:10:46.098490 21728 tablet_copy_client.cc:323] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.164.129:33203
I20250622 20:10:46.111515 21523 tablet_copy_service.cc:140] P d85aa688d5bb48ceac0f2645bcfb7ad2: Received BeginTabletCopySession request for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 from peer 519cf2f27cab42d6a2f63949207804b7 ({username='slave'} at 127.17.164.130:42253)
I20250622 20:10:46.111949 21523 tablet_copy_service.cc:161] P d85aa688d5bb48ceac0f2645bcfb7ad2: Beginning new tablet copy session on tablet 3f0e9bf7b0d0491d8fdc4be911914da5 from peer 519cf2f27cab42d6a2f63949207804b7 at {username='slave'} at 127.17.164.130:42253: session id = 519cf2f27cab42d6a2f63949207804b7-3f0e9bf7b0d0491d8fdc4be911914da5
I20250622 20:10:46.116597 21523 tablet_copy_source_session.cc:215] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 20:10:46.119794 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0e9bf7b0d0491d8fdc4be911914da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:46.129662 21728 tablet_copy_client.cc:806] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Starting download of 0 data blocks...
I20250622 20:10:46.130218 21728 tablet_copy_client.cc:670] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Starting download of 1 WAL segments...
I20250622 20:10:46.135203 21728 tablet_copy_client.cc:538] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 20:10:46.140396 21728 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:46.224488 21728 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:46.225204 21728 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:10:46.225654 21728 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.085s user 0.079s sys 0.008s
I20250622 20:10:46.227303 21728 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:46.227725 21728 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: LEARNER
I20250622 20:10:46.228175 21728 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:46.231117 21728 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250622 20:10:46.232993 21523 tablet_copy_service.cc:342] P d85aa688d5bb48ceac0f2645bcfb7ad2: Request end of tablet copy session 519cf2f27cab42d6a2f63949207804b7-3f0e9bf7b0d0491d8fdc4be911914da5 received from {username='slave'} at 127.17.164.130:42253
I20250622 20:10:46.233619 21523 tablet_copy_service.cc:434] P d85aa688d5bb48ceac0f2645bcfb7ad2: ending tablet copy session 519cf2f27cab42d6a2f63949207804b7-3f0e9bf7b0d0491d8fdc4be911914da5 on tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with peer 519cf2f27cab42d6a2f63949207804b7
I20250622 20:10:46.507850 21648 raft_consensus.cc:1215] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250622 20:10:45.394645 21715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:45.394790 21713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:45.395635 21702 server_base.cc:1048] running on GCE node
I20250622 20:10:46.533809 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:46.535990 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:46.537338 21702 hybrid_clock.cc:648] HybridClock initialized: now 1750623046537312 us; error 41 us; skew 500 ppm
I20250622 20:10:46.538075 21702 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:46.548990 21702 webserver.cc:469] Webserver started at http://127.17.164.131:34049/ using document root <none> and password file <none>
I20250622 20:10:46.549913 21702 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:46.550137 21702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:46.557458 21702 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250622 20:10:46.561625 21739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:46.562577 21702 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250622 20:10:46.562846 21702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e51141a4c764f8db3e1f7a2395eaa97"
format_stamp: "Formatted at 2025-06-22 20:10:33 on dist-test-slave-9mkf"
I20250622 20:10:46.564694 21702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:46.614080 21702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:46.615458 21702 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:46.615839 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:46.618294 21702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:46.623401 21746 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:10:46.630462 21702 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:10:46.630676 21702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250622 20:10:46.630924 21702 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:10:46.635100 21702 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:10:46.635282 21702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250622 20:10:46.635582 21746 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap starting.
I20250622 20:10:46.692646 21746 log.cc:826] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:46.800478 21702 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:40057
I20250622 20:10:46.800729 21853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:40057 every 8 connection(s)
I20250622 20:10:46.802922 21702 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:10:46.811664 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 21702
I20250622 20:10:46.813930 21746 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:46.815040 21746 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap complete.
I20250622 20:10:46.816831 21746 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent bootstrapping tablet: real 0.182s user 0.148s sys 0.032s
I20250622 20:10:46.830155 21854 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:46.830612 21854 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:46.831621 21854 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:46.832437 21746 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:46.834491 21746 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Initialized, Role: FOLLOWER
I20250622 20:10:46.834877 21360 ts_manager.cc:194] Registered new tserver with Master: 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:10:46.835135 21746 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:46.835554 21746 raft_consensus.cc:397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:46.835798 21746 raft_consensus.cc:491] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:46.836035 21746 raft_consensus.cc:3058] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:10:46.837110 21360 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:57607
I20250622 20:10:46.841046 21746 raft_consensus.cc:513] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:46.841594 21746 leader_election.cc:304] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e51141a4c764f8db3e1f7a2395eaa97; no voters:
I20250622 20:10:46.842998 21746 leader_election.cc:290] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:10:46.843729 21860 raft_consensus.cc:2802] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:10:46.845875 21854 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:10:46.846792 21746 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent starting tablet: real 0.030s user 0.028s sys 0.000s
I20250622 20:10:46.846832 21860 raft_consensus.cc:695] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: Becoming Leader. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Running, Role: LEADER
I20250622 20:10:46.847687 21860 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } }
I20250622 20:10:46.848562 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:10:46.852931 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250622 20:10:46.856016 18066 ts_itest-base.cc:209] found only 0 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" }
I20250622 20:10:46.858016 21360 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: term changed from 0 to 2, leader changed from <none> to 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131), VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) added. New cstate: current_term: 2 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } }
I20250622 20:10:46.881479 21809 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } }
I20250622 20:10:46.884732 21862 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index -1 to 11, NON_VOTER d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } }
I20250622 20:10:46.892591 21346 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 20:10:46.894652 21741 consensus_peers.cc:487] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 -> Peer d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203): Couldn't send request to peer d85aa688d5bb48ceac0f2645bcfb7ad2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84df4d7d4a0c46bd95bd8e7f8a6863ff. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:46.895227 21360 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: config changed from index -1 to 11, NON_VOTER d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) added. New cstate: current_term: 2 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 20:10:46.903750 21809 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:46.906838 21860 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } } }
W20250622 20:10:46.908828 21741 consensus_peers.cc:487] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 -> Peer d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203): Couldn't send request to peer d85aa688d5bb48ceac0f2645bcfb7ad2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84df4d7d4a0c46bd95bd8e7f8a6863ff. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:46.913059 21346 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250622 20:10:46.916641 21743 consensus_peers.cc:487] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 -> Peer 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337): Couldn't send request to peer 519cf2f27cab42d6a2f63949207804b7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84df4d7d4a0c46bd95bd8e7f8a6863ff. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:46.916744 21360 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: config changed from index 11 to 12, NON_VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) added. New cstate: current_term: 2 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 20:10:46.943068 21734 raft_consensus.cc:1062] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: attempting to promote NON_VOTER 519cf2f27cab42d6a2f63949207804b7 to VOTER
I20250622 20:10:46.944478 21734 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:10:46.948105 21648 raft_consensus.cc:1273] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250622 20:10:46.949270 21733 consensus_queue.cc:1035] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250622 20:10:46.954797 21734 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:46.956017 21648 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:46.962987 21360 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index 9 to 10, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 20:10:46.972851 21503 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } }
I20250622 20:10:46.977950 21648 raft_consensus.cc:1273] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250622 20:10:46.979159 21733 consensus_queue.cc:1035] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250622 20:10:46.984794 21721 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } } }
I20250622 20:10:46.985934 21648 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } } }
W20250622 20:10:46.991658 21435 consensus_peers.cc:487] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 -> Peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Couldn't send request to peer 9e51141a4c764f8db3e1f7a2395eaa97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f0e9bf7b0d0491d8fdc4be911914da5. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:10:46.993427 21346 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250622 20:10:46.996178 21360 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index 10 to 11, NON_VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) added. New cstate: current_term: 2 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250622 20:10:47.463049 21877 ts_tablet_manager.cc:927] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Initiating tablet copy from peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:10:47.464152 21877 tablet_copy_client.cc:323] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Beginning tablet copy session from remote peer at address 127.17.164.131:40057
I20250622 20:10:47.472227 21829 tablet_copy_service.cc:140] P 9e51141a4c764f8db3e1f7a2395eaa97: Received BeginTabletCopySession request for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff from peer 519cf2f27cab42d6a2f63949207804b7 ({username='slave'} at 127.17.164.130:36513)
I20250622 20:10:47.472605 21829 tablet_copy_service.cc:161] P 9e51141a4c764f8db3e1f7a2395eaa97: Beginning new tablet copy session on tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff from peer 519cf2f27cab42d6a2f63949207804b7 at {username='slave'} at 127.17.164.130:36513: session id = 519cf2f27cab42d6a2f63949207804b7-84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:10:47.476855 21829 tablet_copy_source_session.cc:215] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 20:10:47.479359 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:47.488183 21877 tablet_copy_client.cc:806] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Starting download of 0 data blocks...
I20250622 20:10:47.488584 21877 tablet_copy_client.cc:670] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Starting download of 1 WAL segments...
I20250622 20:10:47.491853 21877 tablet_copy_client.cc:538] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 20:10:47.497159 21877 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:47.516542 21881 ts_tablet_manager.cc:927] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Initiating tablet copy from peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:10:47.518056 21881 tablet_copy_client.cc:323] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: tablet copy: Beginning tablet copy session from remote peer at address 127.17.164.131:40057
I20250622 20:10:47.519387 21829 tablet_copy_service.cc:140] P 9e51141a4c764f8db3e1f7a2395eaa97: Received BeginTabletCopySession request for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff from peer d85aa688d5bb48ceac0f2645bcfb7ad2 ({username='slave'} at 127.17.164.129:50919)
I20250622 20:10:47.519754 21829 tablet_copy_service.cc:161] P 9e51141a4c764f8db3e1f7a2395eaa97: Beginning new tablet copy session on tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff from peer d85aa688d5bb48ceac0f2645bcfb7ad2 at {username='slave'} at 127.17.164.129:50919: session id = d85aa688d5bb48ceac0f2645bcfb7ad2-84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:10:47.524364 21829 tablet_copy_source_session.cc:215] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 20:10:47.525343 21884 ts_tablet_manager.cc:927] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Initiating tablet copy from peer d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203)
I20250622 20:10:47.527696 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:47.527853 21884 tablet_copy_client.cc:323] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: tablet copy: Beginning tablet copy session from remote peer at address 127.17.164.129:33203
I20250622 20:10:47.529213 21523 tablet_copy_service.cc:140] P d85aa688d5bb48ceac0f2645bcfb7ad2: Received BeginTabletCopySession request for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 from peer 9e51141a4c764f8db3e1f7a2395eaa97 ({username='slave'} at 127.17.164.131:59847)
I20250622 20:10:47.529584 21523 tablet_copy_service.cc:161] P d85aa688d5bb48ceac0f2645bcfb7ad2: Beginning new tablet copy session on tablet 3f0e9bf7b0d0491d8fdc4be911914da5 from peer 9e51141a4c764f8db3e1f7a2395eaa97 at {username='slave'} at 127.17.164.131:59847: session id = 9e51141a4c764f8db3e1f7a2395eaa97-3f0e9bf7b0d0491d8fdc4be911914da5
I20250622 20:10:47.533334 21523 tablet_copy_source_session.cc:215] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Tablet Copy: opened 0 blocks and 1 log segments
I20250622 20:10:47.535615 21884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0e9bf7b0d0491d8fdc4be911914da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:10:47.542176 21881 tablet_copy_client.cc:806] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: tablet copy: Starting download of 0 data blocks...
I20250622 20:10:47.542706 21881 tablet_copy_client.cc:670] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: tablet copy: Starting download of 1 WAL segments...
I20250622 20:10:47.546677 21881 tablet_copy_client.cc:538] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 20:10:47.548498 21884 tablet_copy_client.cc:806] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: tablet copy: Starting download of 0 data blocks...
I20250622 20:10:47.549028 21884 tablet_copy_client.cc:670] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: tablet copy: Starting download of 1 WAL segments...
I20250622 20:10:47.553133 21884 tablet_copy_client.cc:538] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250622 20:10:47.553777 21881 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap starting.
I20250622 20:10:47.561143 21884 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap starting.
I20250622 20:10:47.600732 21877 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:47.601339 21877 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:10:47.601799 21877 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.105s user 0.092s sys 0.012s
I20250622 20:10:47.603789 21877 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:47.604450 21877 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: LEARNER
I20250622 20:10:47.604892 21877 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:47.607946 21877 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250622 20:10:47.609421 21829 tablet_copy_service.cc:342] P 9e51141a4c764f8db3e1f7a2395eaa97: Request end of tablet copy session 519cf2f27cab42d6a2f63949207804b7-84df4d7d4a0c46bd95bd8e7f8a6863ff received from {username='slave'} at 127.17.164.130:36513
I20250622 20:10:47.609820 21829 tablet_copy_service.cc:434] P 9e51141a4c764f8db3e1f7a2395eaa97: ending tablet copy session 519cf2f27cab42d6a2f63949207804b7-84df4d7d4a0c46bd95bd8e7f8a6863ff on tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with peer 519cf2f27cab42d6a2f63949207804b7
I20250622 20:10:47.675086 21884 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:47.675717 21884 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap complete.
I20250622 20:10:47.676172 21884 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent bootstrapping tablet: real 0.115s user 0.099s sys 0.012s
I20250622 20:10:47.677882 21884 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } }
I20250622 20:10:47.678364 21884 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Initialized, Role: LEARNER
I20250622 20:10:47.678789 21884 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: NON_VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: true } }
I20250622 20:10:47.680048 21884 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 20:10:47.681461 21523 tablet_copy_service.cc:342] P d85aa688d5bb48ceac0f2645bcfb7ad2: Request end of tablet copy session 9e51141a4c764f8db3e1f7a2395eaa97-3f0e9bf7b0d0491d8fdc4be911914da5 received from {username='slave'} at 127.17.164.131:59847
I20250622 20:10:47.681742 21523 tablet_copy_service.cc:434] P d85aa688d5bb48ceac0f2645bcfb7ad2: ending tablet copy session 9e51141a4c764f8db3e1f7a2395eaa97-3f0e9bf7b0d0491d8fdc4be911914da5 on tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with peer 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:10:47.690546 21881 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:47.691084 21881 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap complete.
I20250622 20:10:47.691455 21881 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent bootstrapping tablet: real 0.138s user 0.115s sys 0.020s
I20250622 20:10:47.692955 21881 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:47.693339 21881 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Initialized, Role: LEARNER
I20250622 20:10:47.693657 21881 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: NON_VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: true } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:47.695129 21881 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250622 20:10:47.696339 21829 tablet_copy_service.cc:342] P 9e51141a4c764f8db3e1f7a2395eaa97: Request end of tablet copy session d85aa688d5bb48ceac0f2645bcfb7ad2-84df4d7d4a0c46bd95bd8e7f8a6863ff received from {username='slave'} at 127.17.164.129:50919
I20250622 20:10:47.696579 21829 tablet_copy_service.cc:434] P 9e51141a4c764f8db3e1f7a2395eaa97: ending tablet copy session d85aa688d5bb48ceac0f2645bcfb7ad2-84df4d7d4a0c46bd95bd8e7f8a6863ff on tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with peer d85aa688d5bb48ceac0f2645bcfb7ad2
I20250622 20:10:47.860267 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d85aa688d5bb48ceac0f2645bcfb7ad2 to finish bootstrapping
I20250622 20:10:47.872438 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 519cf2f27cab42d6a2f63949207804b7 to finish bootstrapping
I20250622 20:10:47.883544 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9e51141a4c764f8db3e1f7a2395eaa97 to finish bootstrapping
I20250622 20:10:47.886607 21347 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250622 20:10:47.929127 21503 raft_consensus.cc:1215] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250622 20:10:48.023187 21648 raft_consensus.cc:1215] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250622 20:10:48.127502 21809 raft_consensus.cc:1215] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250622 20:10:48.157913 21628 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:10:48.169492 21789 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:10:48.170349 21483 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3594a994e5b9499d866dfc3d04585dfd | 127.17.164.190:33417 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:40137 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
519cf2f27cab42d6a2f63949207804b7 | 127.17.164.130:35337 | HEALTHY | <none> | 1 | 0
9e51141a4c764f8db3e1f7a2395eaa97 | 127.17.164.131:40057 | HEALTHY | <none> | 1 | 0
d85aa688d5bb48ceac0f2645bcfb7ad2 | 127.17.164.129:33203 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.164.129 | experimental | 127.17.164.129:33203
local_ip_for_outbound_sockets | 127.17.164.130 | experimental | 127.17.164.130:35337
local_ip_for_outbound_sockets | 127.17.164.131 | experimental | 127.17.164.131:40057
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.17.164.129:33203
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.17.164.130:35337
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.17.164.131:40057
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:40137 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 20:10:48.400305 18066 log_verifier.cc:126] Checking tablet 3f0e9bf7b0d0491d8fdc4be911914da5
I20250622 20:10:48.419826 21900 raft_consensus.cc:1062] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: attempting to promote NON_VOTER d85aa688d5bb48ceac0f2645bcfb7ad2 to VOTER
I20250622 20:10:48.421380 21900 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } }
I20250622 20:10:48.426223 21503 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEARNER]: Refusing update from remote peer 9e51141a4c764f8db3e1f7a2395eaa97: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250622 20:10:48.426446 21648 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Refusing update from remote peer 9e51141a4c764f8db3e1f7a2395eaa97: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250622 20:10:48.427392 21900 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250622 20:10:48.428062 21876 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250622 20:10:48.434686 21899 raft_consensus.cc:1025] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: attempt to promote peer 519cf2f27cab42d6a2f63949207804b7: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250622 20:10:48.435106 21876 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } } }
I20250622 20:10:48.436551 21503 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } } }
I20250622 20:10:48.449164 21648 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Committing config change with OpId 2.13: config changed from index 12 to 13, d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } } }
I20250622 20:10:48.449682 21358 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: config changed from index 12 to 13, d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: NON_VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250622 20:10:48.460337 21876 raft_consensus.cc:1062] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: attempting to promote NON_VOTER 519cf2f27cab42d6a2f63949207804b7 to VOTER
I20250622 20:10:48.462324 21876 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:10:48.467943 21648 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 LEARNER]: Refusing update from remote peer 9e51141a4c764f8db3e1f7a2395eaa97: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250622 20:10:48.469166 21503 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Refusing update from remote peer 9e51141a4c764f8db3e1f7a2395eaa97: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20250622 20:10:48.469242 21900 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250622 20:10:48.470649 21900 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250622 20:10:48.475047 21900 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:48.477007 21647 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:48.477955 21503 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:48.490332 21360 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 reported cstate change: config changed from index 13 to 14, 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 20:10:48.510706 18066 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 3f0e9bf7b0d0491d8fdc4be911914da5
I20250622 20:10:48.511023 18066 log_verifier.cc:126] Checking tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:10:48.596087 18066 log_verifier.cc:177] Verified matching terms for 14 ops in tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:10:48.596411 18066 log_verifier.cc:126] Checking tablet bd5bfc94658b418db7da4d1d4a97c5d0
I20250622 20:10:48.620075 18066 log_verifier.cc:177] Verified matching terms for 7 ops in tablet bd5bfc94658b418db7da4d1d4a97c5d0
I20250622 20:10:48.620443 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 21328
I20250622 20:10:48.647161 18066 minidump.cc:252] Setting minidump size limit to 20M
I20250622 20:10:48.648348 18066 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:48.649271 18066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:48.658427 21928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:48.704905 21914 raft_consensus.cc:1062] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: attempting to promote NON_VOTER 9e51141a4c764f8db3e1f7a2395eaa97 to VOTER
I20250622 20:10:48.707427 21914 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:48.713995 21809 raft_consensus.cc:1273] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 LEARNER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250622 20:10:48.717406 21647 raft_consensus.cc:1273] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250622 20:10:48.718690 21914 consensus_queue.cc:1035] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250622 20:10:48.720098 21936 consensus_queue.cc:1035] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250622 20:10:48.734129 21914 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } } }
I20250622 20:10:48.738571 21647 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } } }
I20250622 20:10:48.754547 21809 raft_consensus.cc:2953] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } } }
W20250622 20:10:48.772516 21698 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
W20250622 20:10:48.772814 21854 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
W20250622 20:10:48.776117 21548 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:33417 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:33417: connect: Connection refused (error 111)
W20250622 20:10:48.659912 21931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:48.658746 21929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:48.782897 18066 server_base.cc:1048] running on GCE node
I20250622 20:10:48.783984 18066 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250622 20:10:48.784215 18066 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250622 20:10:48.784394 18066 hybrid_clock.cc:648] HybridClock initialized: now 1750623048784376 us; error 0 us; skew 500 ppm
I20250622 20:10:48.785090 18066 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:48.787825 18066 webserver.cc:469] Webserver started at http://0.0.0.0:38787/ using document root <none> and password file <none>
I20250622 20:10:48.788620 18066 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:48.788817 18066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:48.793582 18066 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20250622 20:10:48.797072 21945 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:48.797998 18066 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250622 20:10:48.798278 18066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:48.799755 18066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:48.813759 18066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:48.815088 18066 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:48.815500 18066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:48.823760 18066 sys_catalog.cc:263] Verifying existing consensus state
W20250622 20:10:48.826969 18066 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.17.164.190:33417 exists but no master address supplied!
I20250622 20:10:48.828918 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap starting.
I20250622 20:10:48.867733 18066 log.cc:826] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:48.924938 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:48.925668 18066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap complete.
I20250622 20:10:48.938234 18066 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:48.938757 18066 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Initialized, Role: FOLLOWER
I20250622 20:10:48.939384 18066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:48.939980 18066 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:48.940307 18066 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:48.940613 18066 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 3 FOLLOWER]: Advancing to term 4
I20250622 20:10:48.945514 18066 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:48.946130 18066 leader_election.cc:304] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3594a994e5b9499d866dfc3d04585dfd; no voters:
I20250622 20:10:48.947186 18066 leader_election.cc:290] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 4 election: Requested vote from peers
I20250622 20:10:48.947443 21952 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 4 FOLLOWER]: Leader election won for term 4
I20250622 20:10:48.949270 21952 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 4 LEADER]: Becoming Leader. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Running, Role: LEADER
I20250622 20:10:48.949975 21952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:48.956689 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:48.957068 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3594a994e5b9499d866dfc3d04585dfd. Latest consensus state: current_term: 4 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:48.957377 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:48.957547 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:48.991077 18066 tablet_replica.cc:331] stopping tablet replica
I20250622 20:10:48.991737 18066 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 4 LEADER]: Raft consensus shutting down.
I20250622 20:10:48.992205 18066 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 4 FOLLOWER]: Raft consensus is shut down!
I20250622 20:10:48.995260 18066 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 20:10:48.995836 18066 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 20:10:49.026177 18066 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250622 20:10:50.964504 21544 debug-util.cc:398] Leaking SignalData structure 0x7b08000cad00 after lost signal to thread 21418
W20250622 20:10:50.966107 21544 debug-util.cc:398] Leaking SignalData structure 0x7b08000cabc0 after lost signal to thread 21547
I20250622 20:10:54.376581 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 21398
I20250622 20:10:54.402632 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 21549
I20250622 20:10:54.428579 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 21702
I20250622 20:10:54.456923 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--webserver_interface=127.17.164.190
--webserver_port=38957
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:33417 with env {}
W20250622 20:10:54.738692 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:54.739271 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:54.739729 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:54.769831 22026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:10:54.770166 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:54.770458 22026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:10:54.770718 22026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:10:54.804322 22026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:33417
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=38957
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:54.805671 22026 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:54.807220 22026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:54.822109 22032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:54.822371 22035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:54.823647 22026 server_base.cc:1048] running on GCE node
W20250622 20:10:54.822364 22033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:55.950076 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:55.952504 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:55.953822 22026 hybrid_clock.cc:648] HybridClock initialized: now 1750623055953811 us; error 50 us; skew 500 ppm
I20250622 20:10:55.954617 22026 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:55.960948 22026 webserver.cc:469] Webserver started at http://127.17.164.190:38957/ using document root <none> and password file <none>
I20250622 20:10:55.961802 22026 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:55.962038 22026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:55.969641 22026 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250622 20:10:55.973891 22042 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:55.974879 22026 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 20:10:55.975173 22026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3594a994e5b9499d866dfc3d04585dfd"
format_stamp: "Formatted at 2025-06-22 20:10:28 on dist-test-slave-9mkf"
I20250622 20:10:55.976966 22026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:56.027261 22026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:56.028748 22026 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:56.029187 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:56.098917 22026 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:33417
I20250622 20:10:56.099041 22093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:33417 every 8 connection(s)
I20250622 20:10:56.102298 22026 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:10:56.105540 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22026
I20250622 20:10:56.107614 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:33203
--local_ip_for_outbound_sockets=127.17.164.129
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=46875
--webserver_interface=127.17.164.129
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:56.116797 22094 sys_catalog.cc:263] Verifying existing consensus state
I20250622 20:10:56.121438 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap starting.
I20250622 20:10:56.131018 22094 log.cc:826] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:56.204492 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:56.205257 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Bootstrap complete.
I20250622 20:10:56.223667 22094 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:56.225620 22094 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Initialized, Role: FOLLOWER
I20250622 20:10:56.226323 22094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:56.226729 22094 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:10:56.226958 22094 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:10:56.227191 22094 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 5 FOLLOWER]: Advancing to term 6
I20250622 20:10:56.232138 22094 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:56.232789 22094 leader_election.cc:304] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3594a994e5b9499d866dfc3d04585dfd; no voters:
I20250622 20:10:56.234957 22094 leader_election.cc:290] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [CANDIDATE]: Term 6 election: Requested vote from peers
I20250622 20:10:56.235423 22098 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 6 FOLLOWER]: Leader election won for term 6
I20250622 20:10:56.238616 22098 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [term 6 LEADER]: Becoming Leader. State: Replica: 3594a994e5b9499d866dfc3d04585dfd, State: Running, Role: LEADER
I20250622 20:10:56.239424 22098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } }
I20250622 20:10:56.239943 22094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:10:56.249428 22100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3594a994e5b9499d866dfc3d04585dfd. Latest consensus state: current_term: 6 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:56.250130 22100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:56.252351 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "3594a994e5b9499d866dfc3d04585dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3594a994e5b9499d866dfc3d04585dfd" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 33417 } } }
I20250622 20:10:56.252928 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd [sys.catalog]: This master's current role is: LEADER
I20250622 20:10:56.264595 22105 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:10:56.280551 22105 catalog_manager.cc:671] Loaded metadata for table TestTable [id=7c77c458dbd6493b9d117794dc6c1f9d]
I20250622 20:10:56.282800 22105 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=810e070ace744ae1b962c5c441e20eb6]
I20250622 20:10:56.284922 22105 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=84c5c80c9eb545c9bcfd92175c478c37]
I20250622 20:10:56.295392 22105 tablet_loader.cc:96] loaded metadata for tablet 3f0e9bf7b0d0491d8fdc4be911914da5 (table TestTable1 [id=84c5c80c9eb545c9bcfd92175c478c37])
I20250622 20:10:56.297081 22105 tablet_loader.cc:96] loaded metadata for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff (table TestTable [id=7c77c458dbd6493b9d117794dc6c1f9d])
I20250622 20:10:56.298702 22105 tablet_loader.cc:96] loaded metadata for tablet bd5bfc94658b418db7da4d1d4a97c5d0 (table TestTable2 [id=810e070ace744ae1b962c5c441e20eb6])
I20250622 20:10:56.301044 22105 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:10:56.306376 22105 catalog_manager.cc:1261] Loaded cluster ID: acd7998f65b7440fac8aee051820706b
I20250622 20:10:56.306658 22105 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:10:56.315865 22105 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:10:56.321271 22105 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 3594a994e5b9499d866dfc3d04585dfd: Loaded TSK: 0
I20250622 20:10:56.322728 22105 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 20:10:56.437924 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:56.438424 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:56.438921 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:56.468770 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:56.469607 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:10:56.503500 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:33203
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=46875
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:56.504863 22096 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:56.506403 22096 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:56.523285 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:56.526683 22096 server_base.cc:1048] running on GCE node
W20250622 20:10:56.526376 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:56.525373 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:57.651135 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:57.653889 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:57.655411 22096 hybrid_clock.cc:648] HybridClock initialized: now 1750623057655347 us; error 63 us; skew 500 ppm
I20250622 20:10:57.656476 22096 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:57.664096 22096 webserver.cc:469] Webserver started at http://127.17.164.129:46875/ using document root <none> and password file <none>
I20250622 20:10:57.665318 22096 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:57.665580 22096 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:57.675700 22096 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 20:10:57.681470 22131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:57.682627 22096 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250622 20:10:57.682991 22096 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2"
format_stamp: "Formatted at 2025-06-22 20:10:30 on dist-test-slave-9mkf"
I20250622 20:10:57.685688 22096 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:57.750484 22096 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:57.752266 22096 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:57.752801 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:57.755815 22096 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:57.762910 22138 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 20:10:57.773715 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 20:10:57.774003 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250622 20:10:57.774356 22096 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 20:10:57.781600 22138 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap starting.
I20250622 20:10:57.785907 22096 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 20:10:57.786159 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.008s sys 0.003s
I20250622 20:10:57.842857 22138 log.cc:826] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:57.970371 22096 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:33203
I20250622 20:10:57.970535 22245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:33203 every 8 connection(s)
I20250622 20:10:57.972966 22096 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:10:57.982995 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22096
I20250622 20:10:57.984318 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:35337
--local_ip_for_outbound_sockets=127.17.164.130
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=37081
--webserver_interface=127.17.164.130
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:57.985555 22138 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:57.986608 22138 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap complete.
I20250622 20:10:57.988816 22138 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent bootstrapping tablet: real 0.208s user 0.158s sys 0.044s
I20250622 20:10:58.011837 22138 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:58.015029 22138 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Initialized, Role: FOLLOWER
I20250622 20:10:58.016005 22138 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:58.026243 22138 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent starting tablet: real 0.037s user 0.018s sys 0.012s
I20250622 20:10:58.026567 22246 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:58.027115 22138 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap starting.
I20250622 20:10:58.027117 22246 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:58.028569 22246 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:58.034233 22059 ts_manager.cc:194] Registered new tserver with Master: d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129:33203)
I20250622 20:10:58.039065 22059 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index 11 to 12, leader changed from d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) to <none>, 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } } }
I20250622 20:10:58.041021 22059 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index -1 to 14, term changed from 0 to 2, VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) added, VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) added, VOTER d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129) added. New cstate: current_term: 2 committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:10:58.110090 22059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:49729
I20250622 20:10:58.115199 22246 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:10:58.204623 22138 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:58.205375 22138 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Bootstrap complete.
I20250622 20:10:58.206503 22138 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent bootstrapping tablet: real 0.180s user 0.150s sys 0.017s
I20250622 20:10:58.208024 22138 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:10:58.208530 22138 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Initialized, Role: FOLLOWER
I20250622 20:10:58.209125 22138 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:10:58.210605 22138 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250622 20:10:58.358991 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:10:58.359506 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:10:58.360064 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:10:58.390341 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:10:58.391177 22250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:10:58.424649 22250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:35337
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=37081
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:10:58.425920 22250 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:10:58.427453 22250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:10:58.442538 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:10:59.405745 22268 raft_consensus.cc:491] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:59.406495 22268 raft_consensus.cc:513] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:59.410257 22268 leader_election.cc:290] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337), 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
W20250622 20:10:59.414289 22133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
W20250622 20:10:59.419808 22133 leader_election.cc:336] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
W20250622 20:10:59.424279 22135 leader_election.cc:336] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337): Network error: Client connection negotiation failed: client connection to 127.17.164.130:35337: connect: Connection refused (error 111)
I20250622 20:10:59.424690 22135 leader_election.cc:304] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d85aa688d5bb48ceac0f2645bcfb7ad2; no voters: 519cf2f27cab42d6a2f63949207804b7, 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:10:59.425547 22268 raft_consensus.cc:2747] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250622 20:10:59.502615 22268 raft_consensus.cc:491] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:10:59.503254 22268 raft_consensus.cc:513] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:10:59.505735 22268 leader_election.cc:290] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057), 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
W20250622 20:10:59.510237 22133 leader_election.cc:336] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
W20250622 20:10:59.511749 22135 leader_election.cc:336] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337): Network error: Client connection negotiation failed: client connection to 127.17.164.130:35337: connect: Connection refused (error 111)
I20250622 20:10:59.512255 22135 leader_election.cc:304] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d85aa688d5bb48ceac0f2645bcfb7ad2; no voters: 519cf2f27cab42d6a2f63949207804b7, 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:10:59.513126 22268 raft_consensus.cc:2747] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250622 20:10:58.442653 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:58.443626 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:10:59.602355 22264 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1159 milliseconds
I20250622 20:10:59.602475 22250 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:10:59.603670 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:10:59.606345 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:10:59.607810 22250 hybrid_clock.cc:648] HybridClock initialized: now 1750623059607786 us; error 42 us; skew 500 ppm
I20250622 20:10:59.608609 22250 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:10:59.619942 22250 webserver.cc:469] Webserver started at http://127.17.164.130:37081/ using document root <none> and password file <none>
I20250622 20:10:59.620800 22250 fs_manager.cc:362] Metadata directory not provided
I20250622 20:10:59.621002 22250 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:10:59.628232 22250 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250622 20:10:59.632848 22276 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:10:59.633845 22250 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:10:59.634156 22250 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "519cf2f27cab42d6a2f63949207804b7"
format_stamp: "Formatted at 2025-06-22 20:10:31 on dist-test-slave-9mkf"
I20250622 20:10:59.635951 22250 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:10:59.691082 22250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:10:59.692431 22250 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:10:59.692845 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:10:59.695366 22250 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:10:59.700842 22283 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250622 20:10:59.715821 22250 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250622 20:10:59.716053 22250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.002s sys 0.001s
I20250622 20:10:59.716356 22250 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250622 20:10:59.721637 22283 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:59.727283 22250 ts_tablet_manager.cc:610] Registered 3 tablets
I20250622 20:10:59.727485 22250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.009s sys 0.000s
I20250622 20:10:59.772151 22283 log.cc:826] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Log is configured to *not* fsync() on all Append() calls
I20250622 20:10:59.889159 22250 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:35337
I20250622 20:10:59.889284 22390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:35337 every 8 connection(s)
I20250622 20:10:59.892462 22250 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:10:59.893220 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22250
I20250622 20:10:59.895223 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:40057
--local_ip_for_outbound_sockets=127.17.164.131
--tserver_master_addrs=127.17.164.190:33417
--webserver_port=34049
--webserver_interface=127.17.164.131
--builtin_ntp_servers=127.17.164.148:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250622 20:10:59.920775 22283 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:10:59.921881 22283 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:10:59.923350 22283 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.202s user 0.163s sys 0.036s
I20250622 20:10:59.933707 22391 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:10:59.934212 22391 heartbeater.cc:461] Registering TS with master...
I20250622 20:10:59.935374 22391 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:10:59.939713 22059 ts_manager.cc:194] Registered new tserver with Master: 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
I20250622 20:10:59.937638 22283 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:59.940193 22283 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: FOLLOWER
I20250622 20:10:59.941159 22283 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:10:59.944905 22283 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.021s user 0.020s sys 0.001s
I20250622 20:10:59.945075 22059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:34495
I20250622 20:10:59.945569 22283 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:10:59.948516 22391 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:11:00.037140 22283 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:11:00.037822 22283 tablet_bootstrap.cc:492] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:11:00.038991 22283 ts_tablet_manager.cc:1397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.094s user 0.081s sys 0.008s
I20250622 20:11:00.040470 22283 raft_consensus.cc:357] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:11:00.040844 22283 raft_consensus.cc:738] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: FOLLOWER
I20250622 20:11:00.041288 22283 consensus_queue.cc:260] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:11:00.041707 22283 raft_consensus.cc:397] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:11:00.041947 22283 raft_consensus.cc:491] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:11:00.042246 22283 raft_consensus.cc:3058] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:11:00.047120 22283 raft_consensus.cc:513] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:11:00.047706 22283 leader_election.cc:304] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7; no voters:
I20250622 20:11:00.048174 22283 leader_election.cc:290] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250622 20:11:00.048424 22396 raft_consensus.cc:2802] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:11:00.050455 22396 raft_consensus.cc:695] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [term 3 LEADER]: Becoming Leader. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Running, Role: LEADER
I20250622 20:11:00.051225 22396 consensus_queue.cc:237] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } }
I20250622 20:11:00.053965 22283 ts_tablet_manager.cc:1428] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20250622 20:11:00.054509 22283 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap starting.
I20250622 20:11:00.070513 22059 catalog_manager.cc:5582] T bd5bfc94658b418db7da4d1d4a97c5d0 P 519cf2f27cab42d6a2f63949207804b7 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "519cf2f27cab42d6a2f63949207804b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } health_report { overall_health: HEALTHY } } }
I20250622 20:11:00.204102 22283 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:11:00.204823 22283 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Bootstrap complete.
I20250622 20:11:00.205929 22283 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Time spent bootstrapping tablet: real 0.152s user 0.132s sys 0.015s
I20250622 20:11:00.207340 22283 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:00.207764 22283 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519cf2f27cab42d6a2f63949207804b7, State: Initialized, Role: FOLLOWER
I20250622 20:11:00.208248 22283 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:00.209640 22283 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250622 20:11:00.237195 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:00.237648 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:00.238106 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:00.269773 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:00.270560 22395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:11:00.304085 22395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:40057
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=34049
--tserver_master_addrs=127.17.164.190:33417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:00.305310 22395 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:00.306834 22395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:00.321869 22412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:01.042882 22418 raft_consensus.cc:491] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:11:01.043638 22418 raft_consensus.cc:513] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:11:01.046424 22418 leader_election.cc:290] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337), 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
W20250622 20:11:01.058362 22133 leader_election.cc:336] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
I20250622 20:11:01.063999 22346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f0e9bf7b0d0491d8fdc4be911914da5" candidate_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "519cf2f27cab42d6a2f63949207804b7" is_pre_election: true
I20250622 20:11:01.064633 22346 raft_consensus.cc:2466] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85aa688d5bb48ceac0f2645bcfb7ad2 in term 2.
I20250622 20:11:01.066075 22135 leader_election.cc:304] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7, d85aa688d5bb48ceac0f2645bcfb7ad2; no voters: 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:11:01.067147 22418 raft_consensus.cc:2802] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 20:11:01.067409 22418 raft_consensus.cc:491] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:11:01.067657 22418 raft_consensus.cc:3058] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:11:01.073701 22418 raft_consensus.cc:513] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:11:01.075119 22418 leader_election.cc:290] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: Requested vote from peers 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337), 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:11:01.076285 22346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f0e9bf7b0d0491d8fdc4be911914da5" candidate_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "519cf2f27cab42d6a2f63949207804b7"
I20250622 20:11:01.076877 22346 raft_consensus.cc:3058] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Advancing to term 3
W20250622 20:11:01.079322 22133 leader_election.cc:336] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
I20250622 20:11:01.081668 22346 raft_consensus.cc:2466] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85aa688d5bb48ceac0f2645bcfb7ad2 in term 3.
I20250622 20:11:01.082545 22135 leader_election.cc:304] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7, d85aa688d5bb48ceac0f2645bcfb7ad2; no voters: 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:11:01.083106 22418 raft_consensus.cc:2802] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:11:01.085038 22418 raft_consensus.cc:695] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 LEADER]: Becoming Leader. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Running, Role: LEADER
I20250622 20:11:01.085886 22418 consensus_queue.cc:237] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:11:01.101359 22059 catalog_manager.cc:5582] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: term changed from 2 to 3, leader changed from <none> to d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129). New cstate: current_term: 3 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250622 20:11:01.139130 22418 raft_consensus.cc:491] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:11:01.139896 22418 raft_consensus.cc:513] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:01.142843 22418 leader_election.cc:290] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057), 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
I20250622 20:11:01.143923 22346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" candidate_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "519cf2f27cab42d6a2f63949207804b7" is_pre_election: true
I20250622 20:11:01.144524 22346 raft_consensus.cc:2466] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85aa688d5bb48ceac0f2645bcfb7ad2 in term 2.
I20250622 20:11:01.145545 22135 leader_election.cc:304] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7, d85aa688d5bb48ceac0f2645bcfb7ad2; no voters:
I20250622 20:11:01.146135 22418 raft_consensus.cc:2802] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 20:11:01.146380 22418 raft_consensus.cc:491] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:11:01.146633 22418 raft_consensus.cc:3058] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 2 FOLLOWER]: Advancing to term 3
W20250622 20:11:01.146986 22133 leader_election.cc:336] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
I20250622 20:11:01.150861 22418 raft_consensus.cc:513] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:01.152251 22418 leader_election.cc:290] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: Requested vote from peers 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057), 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337)
I20250622 20:11:01.153651 22346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" candidate_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" candidate_term: 3 candidate_status { last_received { term: 2 index: 14 } } ignore_live_leader: false dest_uuid: "519cf2f27cab42d6a2f63949207804b7"
I20250622 20:11:01.154147 22346 raft_consensus.cc:3058] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 2 FOLLOWER]: Advancing to term 3
W20250622 20:11:01.156242 22133 leader_election.cc:336] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111)
I20250622 20:11:01.158607 22346 raft_consensus.cc:2466] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85aa688d5bb48ceac0f2645bcfb7ad2 in term 3.
I20250622 20:11:01.159536 22135 leader_election.cc:304] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 519cf2f27cab42d6a2f63949207804b7, d85aa688d5bb48ceac0f2645bcfb7ad2; no voters: 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:11:01.160240 22418 raft_consensus.cc:2802] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 20:11:01.160909 22418 raft_consensus.cc:695] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 LEADER]: Becoming Leader. State: Replica: d85aa688d5bb48ceac0f2645bcfb7ad2, State: Running, Role: LEADER
I20250622 20:11:01.161950 22418 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:01.171518 22059 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: term changed from 2 to 3, leader changed from <none> to d85aa688d5bb48ceac0f2645bcfb7ad2 (127.17.164.129). New cstate: current_term: 3 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250622 20:11:00.322834 22413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:00.323562 22395 server_base.cc:1048] running on GCE node
W20250622 20:11:00.323482 22415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:01.474225 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:01.476876 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:01.478291 22395 hybrid_clock.cc:648] HybridClock initialized: now 1750623061478238 us; error 72 us; skew 500 ppm
I20250622 20:11:01.479054 22395 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:01.485901 22395 webserver.cc:469] Webserver started at http://127.17.164.131:34049/ using document root <none> and password file <none>
I20250622 20:11:01.486730 22395 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:01.486908 22395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:01.494313 22395 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250622 20:11:01.499557 22434 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250622 20:11:01.500142 22133 consensus_peers.cc:487] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 -> Peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Couldn't send request to peer 9e51141a4c764f8db3e1f7a2395eaa97. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 20:11:01.500666 22395 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250622 20:11:01.500989 22395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e51141a4c764f8db3e1f7a2395eaa97"
format_stamp: "Formatted at 2025-06-22 20:10:33 on dist-test-slave-9mkf"
I20250622 20:11:01.502785 22395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:01.508589 22346 raft_consensus.cc:1273] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250622 20:11:01.509902 22418 consensus_queue.cc:1035] T 3f0e9bf7b0d0491d8fdc4be911914da5 P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250622 20:11:01.571918 22395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:01.573369 22395 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:01.573801 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:01.576200 22395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:01.582048 22445 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250622 20:11:01.592927 22395 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250622 20:11:01.593160 22395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250622 20:11:01.593433 22395 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250622 20:11:01.598433 22445 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap starting.
I20250622 20:11:01.600970 22395 ts_tablet_manager.cc:610] Registered 2 tablets
I20250622 20:11:01.601150 22395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.008s sys 0.000s
I20250622 20:11:01.649801 22445 log.cc:826] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Log is configured to *not* fsync() on all Append() calls
W20250622 20:11:01.673743 22133 consensus_peers.cc:487] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 -> Peer 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): Couldn't send request to peer 9e51141a4c764f8db3e1f7a2395eaa97. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.131:40057: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 20:11:01.736256 22445 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:11:01.737062 22445 tablet_bootstrap.cc:492] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap complete.
I20250622 20:11:01.738464 22445 ts_tablet_manager.cc:1397] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent bootstrapping tablet: real 0.140s user 0.105s sys 0.032s
I20250622 20:11:01.753155 22445 raft_consensus.cc:357] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:11:01.756744 22445 raft_consensus.cc:738] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Initialized, Role: FOLLOWER
I20250622 20:11:01.757232 22346 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250622 20:11:01.757973 22445 consensus_queue.cc:260] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } attrs { promote: false } }
I20250622 20:11:01.758992 22423 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250622 20:11:01.763610 22445 ts_tablet_manager.cc:1428] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent starting tablet: real 0.025s user 0.020s sys 0.003s
I20250622 20:11:01.764557 22445 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap starting.
I20250622 20:11:01.799247 22201 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 14, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:01.804248 22346 raft_consensus.cc:1273] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Refusing update from remote peer d85aa688d5bb48ceac0f2645bcfb7ad2: Log matching property violated. Preceding OpId in replica: term: 3 index: 15. Preceding OpId from leader: term: 3 index: 16. (index mismatch)
I20250622 20:11:01.806124 22395 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:40057
I20250622 20:11:01.805917 22418 consensus_queue.cc:1035] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20250622 20:11:01.806382 22556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:40057 every 8 connection(s)
I20250622 20:11:01.811221 22395 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:11:01.811111 22423 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 14 to 16, VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:11:01.817417 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22395
I20250622 20:11:01.817551 22346 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Committing config change with OpId 3.16: config changed from index 14 to 16, VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } } }
I20250622 20:11:01.826143 22044 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 20:11:01.846518 22059 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index 14 to 16, VOTER 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131) evicted. New cstate: current_term: 3 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250622 20:11:01.881609 22059 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff on TS 9e51141a4c764f8db3e1f7a2395eaa97: Not found: failed to reset TS proxy: Could not find TS for UUID 9e51141a4c764f8db3e1f7a2395eaa97
I20250622 20:11:01.882483 22557 heartbeater.cc:344] Connected to a master server at 127.17.164.190:33417
I20250622 20:11:01.883020 22557 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:01.884331 22557 heartbeater.cc:507] Master 127.17.164.190:33417 requested a full tablet report, sending...
I20250622 20:11:01.886296 22201 consensus_queue.cc:237] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 3.16, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } }
I20250622 20:11:01.888934 22423 raft_consensus.cc:2953] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 [term 3 LEADER]: Committing config change with OpId 3.17: config changed from index 16 to 17, VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) evicted. New config: { opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } }
I20250622 20:11:01.895596 22059 ts_manager.cc:194] Registered new tserver with Master: 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057)
I20250622 20:11:01.897286 22044 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 20:11:01.900636 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:11:01.902016 22058 catalog_manager.cc:5582] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P d85aa688d5bb48ceac0f2645bcfb7ad2 reported cstate change: config changed from index 16 to 17, VOTER 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130) evicted. New cstate: current_term: 3 leader_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" committed_config { opid_index: 17 OBSOLETE_local: true peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250622 20:11:01.907092 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250622 20:11:01.911450 18066 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250622 20:11:01.913842 22059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:45491
I20250622 20:11:01.918934 22508 raft_consensus.cc:3058] T 3f0e9bf7b0d0491d8fdc4be911914da5 P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Advancing to term 3
I20250622 20:11:01.932937 22557 heartbeater.cc:499] Master 127.17.164.190:33417 was elected leader, sending a full tablet report...
I20250622 20:11:01.964008 22488 tablet_service.cc:1515] Processing DeleteTablet for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with delete_type TABLET_DATA_TOMBSTONED (TS 9e51141a4c764f8db3e1f7a2395eaa97 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:57318
I20250622 20:11:01.972751 22326 tablet_service.cc:1515] Processing DeleteTablet for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with delete_type TABLET_DATA_TOMBSTONED (TS 519cf2f27cab42d6a2f63949207804b7 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:44732
I20250622 20:11:02.010018 22573 tablet_replica.cc:331] stopping tablet replica
I20250622 20:11:02.010983 22573 raft_consensus.cc:2241] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Raft consensus shutting down.
W20250622 20:11:02.013062 22044 catalog_manager.cc:4908] TS 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): delete failed for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff already in progress: opening tablet
I20250622 20:11:02.014277 22573 raft_consensus.cc:2270] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250622 20:11:02.018074 22573 ts_tablet_manager.cc:1905] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 20:11:02.025488 22445 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:11:02.026172 22445 tablet_bootstrap.cc:492] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Bootstrap complete.
I20250622 20:11:02.027936 22445 ts_tablet_manager.cc:1397] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent bootstrapping tablet: real 0.264s user 0.210s sys 0.031s
I20250622 20:11:02.030452 22445 raft_consensus.cc:357] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:02.031131 22445 raft_consensus.cc:738] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e51141a4c764f8db3e1f7a2395eaa97, State: Initialized, Role: FOLLOWER
I20250622 20:11:02.031733 22445 consensus_queue.cc:260] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "9e51141a4c764f8db3e1f7a2395eaa97" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 40057 } } peers { permanent_uuid: "d85aa688d5bb48ceac0f2645bcfb7ad2" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 33203 } attrs { promote: false } } peers { permanent_uuid: "519cf2f27cab42d6a2f63949207804b7" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 35337 } attrs { promote: false } }
I20250622 20:11:02.033864 22445 ts_tablet_manager.cc:1428] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250622 20:11:02.035315 22573 ts_tablet_manager.cc:1918] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.16
I20250622 20:11:02.035748 22573 log.cc:1199] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 519cf2f27cab42d6a2f63949207804b7: Deleting WAL directory at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/wals/84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:11:02.037837 22046 catalog_manager.cc:4928] TS 519cf2f27cab42d6a2f63949207804b7 (127.17.164.130:35337): tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff (table TestTable [id=7c77c458dbd6493b9d117794dc6c1f9d]) successfully deleted
I20250622 20:11:02.039099 22488 tablet_service.cc:1515] Processing DeleteTablet for tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 14 (current committed config index is 17)) from {username='slave'} at 127.0.0.1:57318
I20250622 20:11:02.040141 22572 tablet_replica.cc:331] stopping tablet replica
I20250622 20:11:02.040979 22572 raft_consensus.cc:2241] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 20:11:02.041529 22572 raft_consensus.cc:2270] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 20:11:02.043967 22572 ts_tablet_manager.cc:1905] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 20:11:02.054056 22572 ts_tablet_manager.cc:1918] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.14
I20250622 20:11:02.054428 22572 log.cc:1199] T 84df4d7d4a0c46bd95bd8e7f8a6863ff P 9e51141a4c764f8db3e1f7a2395eaa97: Deleting WAL directory at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/wals/84df4d7d4a0c46bd95bd8e7f8a6863ff
I20250622 20:11:02.055711 22044 catalog_manager.cc:4928] TS 9e51141a4c764f8db3e1f7a2395eaa97 (127.17.164.131:40057): tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff (table TestTable [id=7c77c458dbd6493b9d117794dc6c1f9d]) successfully deleted
W20250622 20:11:02.916055 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:03.920099 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:04.924314 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:05.927917 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:06.931335 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:07.934999 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:08.938537 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:09.941967 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:10.945325 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:11.948612 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:12.952168 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:13.956080 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:14.960208 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:15.963745 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:16.966991 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:17.970348 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:18.973477 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:19.976857 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250622 20:11:20.479147 22553 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc060 after lost signal to thread 22407
W20250622 20:11:20.480316 22553 debug-util.cc:398] Leaking SignalData structure 0x7b08000cc640 after lost signal to thread 22556
W20250622 20:11:20.980705 18066 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 84df4d7d4a0c46bd95bd8e7f8a6863ff: tablet_id: "84df4d7d4a0c46bd95bd8e7f8a6863ff" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@3/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250622 20:11:21.984884 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22096
I20250622 20:11:22.024907 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22250
I20250622 20:11:22.055598 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22395
I20250622 20:11:22.079388 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22026
2025-06-22T20:11:22Z chronyd exiting
I20250622 20:11:22.126956 18066 test_util.cc:183] -----------------------------------------------
I20250622 20:11:22.127161 18066 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750622963291160-18066-0
[ FAILED ] AdminCliTest.TestRebuildTables (55372 ms)
[----------] 5 tests from AdminCliTest (118766 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250622 20:11:22.132225 18066 test_util.cc:276] Using random seed: -1292750104
I20250622 20:11:22.137105 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:11:22.137271 18066 ts_itest-base.cc:116] --------------
I20250622 20:11:22.137450 18066 ts_itest-base.cc:117] 5 tablet servers
I20250622 20:11:22.137625 18066 ts_itest-base.cc:118] 3 replicas per TS
I20250622 20:11:22.137777 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:11:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:11:22Z Disabled control of system clock
I20250622 20:11:22.179457 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:36471
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:36471
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 20:11:22.468356 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:22.468950 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:22.469399 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:22.498529 22593 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:22.498907 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:11:22.499164 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:22.499401 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:11:22.499662 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:11:22.532828 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:36471
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:36471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:22.534222 22593 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:22.535785 22593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:22.550019 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:22.550341 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:22.550016 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:23.688755 22601 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:23.688903 22593 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:23.692929 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:23.695830 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:23.697240 22593 hybrid_clock.cc:648] HybridClock initialized: now 1750623083697220 us; error 38 us; skew 500 ppm
I20250622 20:11:23.698064 22593 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:23.709604 22593 webserver.cc:469] Webserver started at http://127.17.164.190:33817/ using document root <none> and password file <none>
I20250622 20:11:23.710510 22593 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:23.710704 22593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:23.711100 22593 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:23.715396 22593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e62bc0ea7b4346d98a0772a7c139c142"
format_stamp: "Formatted at 2025-06-22 20:11:23 on dist-test-slave-9mkf"
I20250622 20:11:23.716439 22593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e62bc0ea7b4346d98a0772a7c139c142"
format_stamp: "Formatted at 2025-06-22 20:11:23 on dist-test-slave-9mkf"
I20250622 20:11:23.723501 22593 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250622 20:11:23.728991 22610 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:23.729951 22593 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:11:23.730257 22593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e62bc0ea7b4346d98a0772a7c139c142"
format_stamp: "Formatted at 2025-06-22 20:11:23 on dist-test-slave-9mkf"
I20250622 20:11:23.730582 22593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:23.784310 22593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:23.785789 22593 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:23.786206 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:23.854506 22593 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:36471
I20250622 20:11:23.854568 22661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:36471 every 8 connection(s)
I20250622 20:11:23.857355 22593 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:11:23.858443 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22593
I20250622 20:11:23.859299 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:11:23.863354 22662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:23.887507 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142: Bootstrap starting.
I20250622 20:11:23.892903 22662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:23.894515 22662 log.cc:826] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:23.899042 22662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142: No bootstrap required, opened a new log
I20250622 20:11:23.915449 22662 raft_consensus.cc:357] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } }
I20250622 20:11:23.916080 22662 raft_consensus.cc:383] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:23.916273 22662 raft_consensus.cc:738] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62bc0ea7b4346d98a0772a7c139c142, State: Initialized, Role: FOLLOWER
I20250622 20:11:23.916916 22662 consensus_queue.cc:260] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } }
I20250622 20:11:23.917362 22662 raft_consensus.cc:397] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:11:23.917580 22662 raft_consensus.cc:491] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:11:23.917820 22662 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:23.921819 22662 raft_consensus.cc:513] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } }
I20250622 20:11:23.922461 22662 leader_election.cc:304] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e62bc0ea7b4346d98a0772a7c139c142; no voters:
I20250622 20:11:23.923996 22662 leader_election.cc:290] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:11:23.924675 22667 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:23.926894 22667 raft_consensus.cc:695] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [term 1 LEADER]: Becoming Leader. State: Replica: e62bc0ea7b4346d98a0772a7c139c142, State: Running, Role: LEADER
I20250622 20:11:23.927553 22667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } }
I20250622 20:11:23.928097 22662 sys_catalog.cc:564] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:11:23.939592 22669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e62bc0ea7b4346d98a0772a7c139c142. Latest consensus state: current_term: 1 leader_uuid: "e62bc0ea7b4346d98a0772a7c139c142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } } }
I20250622 20:11:23.941475 22669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:23.939929 22668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e62bc0ea7b4346d98a0772a7c139c142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e62bc0ea7b4346d98a0772a7c139c142" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 36471 } } }
I20250622 20:11:23.942364 22668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:23.944051 22677 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:11:23.954128 22677 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:11:23.971091 22677 catalog_manager.cc:1349] Generated new cluster ID: cc60ee2a8fa54907b197df9ef62e3e1c
I20250622 20:11:23.971402 22677 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:11:23.987756 22677 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:11:23.989172 22677 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:11:24.005235 22677 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e62bc0ea7b4346d98a0772a7c139c142: Generated new TSK 0
I20250622 20:11:24.006263 22677 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:11:24.023080 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250622 20:11:24.326133 22686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:24.326665 22686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:24.327160 22686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:24.359011 22686 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:24.359417 22686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:24.360220 22686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:11:24.396459 22686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:24.397922 22686 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:24.399642 22686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:24.417829 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:24.419027 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:24.417896 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:25.591521 22694 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:25.591970 22686 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:25.595892 22686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:25.598565 22686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:25.600025 22686 hybrid_clock.cc:648] HybridClock initialized: now 1750623085599975 us; error 74 us; skew 500 ppm
I20250622 20:11:25.600881 22686 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:25.607508 22686 webserver.cc:469] Webserver started at http://127.17.164.129:34521/ using document root <none> and password file <none>
I20250622 20:11:25.608402 22686 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:25.608616 22686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:25.609097 22686 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:25.613545 22686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b722af436b1d4e738b4153955160db5f"
format_stamp: "Formatted at 2025-06-22 20:11:25 on dist-test-slave-9mkf"
I20250622 20:11:25.614645 22686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b722af436b1d4e738b4153955160db5f"
format_stamp: "Formatted at 2025-06-22 20:11:25 on dist-test-slave-9mkf"
I20250622 20:11:25.621954 22686 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250622 20:11:25.627972 22702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:25.629035 22686 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250622 20:11:25.629335 22686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b722af436b1d4e738b4153955160db5f"
format_stamp: "Formatted at 2025-06-22 20:11:25 on dist-test-slave-9mkf"
I20250622 20:11:25.629745 22686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:25.691568 22686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:25.693063 22686 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:25.693490 22686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:25.696435 22686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:25.700392 22686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:25.700585 22686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:25.700861 22686 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:25.701027 22686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:25.826474 22686 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:42061
I20250622 20:11:25.826586 22814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:42061 every 8 connection(s)
I20250622 20:11:25.829010 22686 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:11:25.829986 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22686
I20250622 20:11:25.830530 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:11:25.839675 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 20:11:25.886654 22815 heartbeater.cc:344] Connected to a master server at 127.17.164.190:36471
I20250622 20:11:25.887180 22815 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:25.888465 22815 heartbeater.cc:507] Master 127.17.164.190:36471 requested a full tablet report, sending...
I20250622 20:11:25.891682 22627 ts_manager.cc:194] Registered new tserver with Master: b722af436b1d4e738b4153955160db5f (127.17.164.129:42061)
I20250622 20:11:25.894161 22627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:33475
W20250622 20:11:26.149780 22819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:26.150270 22819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:26.150758 22819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:26.180294 22819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:26.180744 22819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:26.181530 22819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:11:26.216471 22819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:26.217825 22819 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:26.219401 22819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:26.235050 22826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:26.883497 22658 debug-util.cc:398] Leaking SignalData structure 0x7b080006f5a0 after lost signal to thread 22594
W20250622 20:11:26.884517 22658 debug-util.cc:398] Leaking SignalData structure 0x7b080006f460 after lost signal to thread 22661
I20250622 20:11:26.898111 22815 heartbeater.cc:499] Master 127.17.164.190:36471 was elected leader, sending a full tablet report...
W20250622 20:11:26.235133 22828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:26.235070 22825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:27.654585 22827 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1417 milliseconds
I20250622 20:11:27.654742 22819 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:27.656359 22819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:27.659272 22819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:27.660833 22819 hybrid_clock.cc:648] HybridClock initialized: now 1750623087660781 us; error 51 us; skew 500 ppm
I20250622 20:11:27.661633 22819 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:27.668862 22819 webserver.cc:469] Webserver started at http://127.17.164.130:36821/ using document root <none> and password file <none>
I20250622 20:11:27.669804 22819 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:27.670009 22819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:27.670454 22819 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:27.674814 22819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ab55f036561a4c6991c7edcabc7b8c19"
format_stamp: "Formatted at 2025-06-22 20:11:27 on dist-test-slave-9mkf"
I20250622 20:11:27.675894 22819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ab55f036561a4c6991c7edcabc7b8c19"
format_stamp: "Formatted at 2025-06-22 20:11:27 on dist-test-slave-9mkf"
I20250622 20:11:27.683526 22819 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250622 20:11:27.690068 22835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:27.691192 22819 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250622 20:11:27.691502 22819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ab55f036561a4c6991c7edcabc7b8c19"
format_stamp: "Formatted at 2025-06-22 20:11:27 on dist-test-slave-9mkf"
I20250622 20:11:27.691820 22819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:27.750569 22819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:27.752133 22819 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:27.752573 22819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:27.755636 22819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:27.759891 22819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:27.760092 22819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:27.760367 22819 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:27.760597 22819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:27.925737 22819 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:42699
I20250622 20:11:27.925920 22947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:42699 every 8 connection(s)
I20250622 20:11:27.928228 22819 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250622 20:11:27.935119 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22819
I20250622 20:11:27.935747 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250622 20:11:27.947443 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 20:11:27.963212 22948 heartbeater.cc:344] Connected to a master server at 127.17.164.190:36471
I20250622 20:11:27.963682 22948 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:27.964864 22948 heartbeater.cc:507] Master 127.17.164.190:36471 requested a full tablet report, sending...
I20250622 20:11:27.966863 22627 ts_manager.cc:194] Registered new tserver with Master: ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699)
I20250622 20:11:27.968132 22627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:51501
W20250622 20:11:28.282843 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:28.283390 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:28.283898 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:28.314113 22952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:28.314538 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:28.315351 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:11:28.349761 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:28.351066 22952 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:28.352694 22952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:28.370090 22958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:28.971099 22948 heartbeater.cc:499] Master 127.17.164.190:36471 was elected leader, sending a full tablet report...
W20250622 20:11:28.370863 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:28.372351 22952 server_base.cc:1048] running on GCE node
W20250622 20:11:28.371609 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:29.511059 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:29.513417 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:29.514784 22952 hybrid_clock.cc:648] HybridClock initialized: now 1750623089514747 us; error 56 us; skew 500 ppm
I20250622 20:11:29.515559 22952 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:29.527927 22952 webserver.cc:469] Webserver started at http://127.17.164.131:34117/ using document root <none> and password file <none>
I20250622 20:11:29.529001 22952 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:29.529219 22952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:29.529767 22952 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:29.534328 22952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1facec447a334a9f9955a0a8212f5d3c"
format_stamp: "Formatted at 2025-06-22 20:11:29 on dist-test-slave-9mkf"
I20250622 20:11:29.535425 22952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1facec447a334a9f9955a0a8212f5d3c"
format_stamp: "Formatted at 2025-06-22 20:11:29 on dist-test-slave-9mkf"
I20250622 20:11:29.542538 22952 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250622 20:11:29.548069 22968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:29.549073 22952 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:11:29.549371 22952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1facec447a334a9f9955a0a8212f5d3c"
format_stamp: "Formatted at 2025-06-22 20:11:29 on dist-test-slave-9mkf"
I20250622 20:11:29.549685 22952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:29.606220 22952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:29.607659 22952 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:29.608059 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:29.610399 22952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:29.614149 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:29.614356 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:29.614598 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:29.614749 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:29.743805 22952 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:34681
I20250622 20:11:29.743875 23080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:34681 every 8 connection(s)
I20250622 20:11:29.747139 22952 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250622 20:11:29.747997 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 22952
I20250622 20:11:29.748399 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250622 20:11:29.762423 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.132:0
--local_ip_for_outbound_sockets=127.17.164.132
--webserver_interface=127.17.164.132
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 20:11:29.791863 23081 heartbeater.cc:344] Connected to a master server at 127.17.164.190:36471
I20250622 20:11:29.792513 23081 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:29.794072 23081 heartbeater.cc:507] Master 127.17.164.190:36471 requested a full tablet report, sending...
I20250622 20:11:29.796495 22627 ts_manager.cc:194] Registered new tserver with Master: 1facec447a334a9f9955a0a8212f5d3c (127.17.164.131:34681)
I20250622 20:11:29.797650 22627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:54087
W20250622 20:11:30.067941 23085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:30.068380 23085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:30.068966 23085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:30.098382 23085 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:30.098749 23085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:30.099512 23085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.132
I20250622 20:11:30.133140 23085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.17.164.132
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.132
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:30.135043 23085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:30.137187 23085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:30.154016 23092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:30.801116 23081 heartbeater.cc:499] Master 127.17.164.190:36471 was elected leader, sending a full tablet report...
W20250622 20:11:30.157694 23094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:30.154325 23091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:31.344799 23093 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1188 milliseconds
I20250622 20:11:31.344906 23085 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:31.346041 23085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:31.348645 23085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:31.350135 23085 hybrid_clock.cc:648] HybridClock initialized: now 1750623091350097 us; error 59 us; skew 500 ppm
I20250622 20:11:31.350926 23085 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:31.356959 23085 webserver.cc:469] Webserver started at http://127.17.164.132:37523/ using document root <none> and password file <none>
I20250622 20:11:31.357811 23085 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:31.358042 23085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:31.358522 23085 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:31.362776 23085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3a91ef9bddc24c4c994624aa9ec1f34c"
format_stamp: "Formatted at 2025-06-22 20:11:31 on dist-test-slave-9mkf"
I20250622 20:11:31.363860 23085 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3a91ef9bddc24c4c994624aa9ec1f34c"
format_stamp: "Formatted at 2025-06-22 20:11:31 on dist-test-slave-9mkf"
I20250622 20:11:31.370606 23085 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250622 20:11:31.375936 23101 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:31.376993 23085 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:11:31.377331 23085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3a91ef9bddc24c4c994624aa9ec1f34c"
format_stamp: "Formatted at 2025-06-22 20:11:31 on dist-test-slave-9mkf"
I20250622 20:11:31.377612 23085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:31.432505 23085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:31.433923 23085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:31.434319 23085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:31.436599 23085 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:31.440435 23085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:31.440656 23085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:31.440932 23085 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:31.441092 23085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:31.569654 23085 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.132:39219
I20250622 20:11:31.569757 23213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.132:39219 every 8 connection(s)
I20250622 20:11:31.572185 23085 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250622 20:11:31.581435 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23085
I20250622 20:11:31.581822 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250622 20:11:31.588013 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.133:0
--local_ip_for_outbound_sockets=127.17.164.133
--webserver_interface=127.17.164.133
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--builtin_ntp_servers=127.17.164.148:40465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250622 20:11:31.593159 23214 heartbeater.cc:344] Connected to a master server at 127.17.164.190:36471
I20250622 20:11:31.593550 23214 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:31.594722 23214 heartbeater.cc:507] Master 127.17.164.190:36471 requested a full tablet report, sending...
I20250622 20:11:31.596542 22627 ts_manager.cc:194] Registered new tserver with Master: 3a91ef9bddc24c4c994624aa9ec1f34c (127.17.164.132:39219)
I20250622 20:11:31.598237 22627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.132:57439
W20250622 20:11:31.899327 23218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:31.899811 23218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:31.900255 23218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:31.932525 23218 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250622 20:11:31.932953 23218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:31.933744 23218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.133
I20250622 20:11:31.967896 23218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:40465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.17.164.133
--webserver_port=0
--tserver_master_addrs=127.17.164.190:36471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.133
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:31.969226 23218 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:31.970775 23218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:31.988552 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:32.601352 23214 heartbeater.cc:499] Master 127.17.164.190:36471 was elected leader, sending a full tablet report...
W20250622 20:11:31.990268 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:33.134474 23226 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:33.134608 23218 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250622 20:11:31.988586 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:33.138926 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:33.140993 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:33.142385 23218 hybrid_clock.cc:648] HybridClock initialized: now 1750623093142334 us; error 56 us; skew 500 ppm
I20250622 20:11:33.143289 23218 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:33.149724 23218 webserver.cc:469] Webserver started at http://127.17.164.133:46247/ using document root <none> and password file <none>
I20250622 20:11:33.150605 23218 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:33.150797 23218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:33.151228 23218 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:33.156008 23218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "57389b5018464dff8e6b2f6f5b359407"
format_stamp: "Formatted at 2025-06-22 20:11:33 on dist-test-slave-9mkf"
I20250622 20:11:33.157099 23218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "57389b5018464dff8e6b2f6f5b359407"
format_stamp: "Formatted at 2025-06-22 20:11:33 on dist-test-slave-9mkf"
I20250622 20:11:33.164126 23218 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250622 20:11:33.169834 23235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:33.170804 23218 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250622 20:11:33.171114 23218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "57389b5018464dff8e6b2f6f5b359407"
format_stamp: "Formatted at 2025-06-22 20:11:33 on dist-test-slave-9mkf"
I20250622 20:11:33.171424 23218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:33.259879 23218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:33.261480 23218 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:33.261934 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:33.264434 23218 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:33.268862 23218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:33.269049 23218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:33.269243 23218 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:33.269376 23218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:33.414214 23218 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.133:39753
I20250622 20:11:33.414309 23347 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.133:39753 every 8 connection(s)
I20250622 20:11:33.416612 23218 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250622 20:11:33.421708 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23218
I20250622 20:11:33.422227 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250622 20:11:33.438122 23348 heartbeater.cc:344] Connected to a master server at 127.17.164.190:36471
I20250622 20:11:33.438494 23348 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:33.440092 23348 heartbeater.cc:507] Master 127.17.164.190:36471 requested a full tablet report, sending...
I20250622 20:11:33.442528 22626 ts_manager.cc:194] Registered new tserver with Master: 57389b5018464dff8e6b2f6f5b359407 (127.17.164.133:39753)
I20250622 20:11:33.443938 22626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.133:33847
I20250622 20:11:33.456631 18066 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 20:11:33.491163 22626 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38154:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 20:11:33.567142 23016 tablet_service.cc:1468] Processing CreateTablet for tablet 9c914b2b829447b08af5f775a7b84f5c (DEFAULT_TABLE table=TestTable [id=a026f83528c2425e8c9c1f335f6eab14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:33.569312 23016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c914b2b829447b08af5f775a7b84f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:33.576853 22883 tablet_service.cc:1468] Processing CreateTablet for tablet 9c914b2b829447b08af5f775a7b84f5c (DEFAULT_TABLE table=TestTable [id=a026f83528c2425e8c9c1f335f6eab14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:33.578771 22883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c914b2b829447b08af5f775a7b84f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:33.582705 23149 tablet_service.cc:1468] Processing CreateTablet for tablet 9c914b2b829447b08af5f775a7b84f5c (DEFAULT_TABLE table=TestTable [id=a026f83528c2425e8c9c1f335f6eab14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:33.584471 23149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c914b2b829447b08af5f775a7b84f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:33.617491 23368 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Bootstrap starting.
I20250622 20:11:33.622084 23369 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Bootstrap starting.
I20250622 20:11:33.622071 23370 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Bootstrap starting.
I20250622 20:11:33.626370 23368 tablet_bootstrap.cc:654] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:33.629338 23368 log.cc:826] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:33.630033 23369 tablet_bootstrap.cc:654] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:33.630020 23370 tablet_bootstrap.cc:654] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:33.632748 23369 log.cc:826] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:33.632747 23370 log.cc:826] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:33.636392 23368 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: No bootstrap required, opened a new log
I20250622 20:11:33.636832 23368 ts_tablet_manager.cc:1397] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20250622 20:11:33.638470 23369 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: No bootstrap required, opened a new log
I20250622 20:11:33.638835 23369 ts_tablet_manager.cc:1397] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250622 20:11:33.639051 23370 tablet_bootstrap.cc:492] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: No bootstrap required, opened a new log
I20250622 20:11:33.639950 23370 ts_tablet_manager.cc:1397] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20250622 20:11:33.658392 23369 raft_consensus.cc:357] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.659308 23369 raft_consensus.cc:383] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:33.659606 23369 raft_consensus.cc:738] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1facec447a334a9f9955a0a8212f5d3c, State: Initialized, Role: FOLLOWER
I20250622 20:11:33.660460 23369 consensus_queue.cc:260] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.664646 23369 ts_tablet_manager.cc:1428] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Time spent starting tablet: real 0.026s user 0.019s sys 0.009s
I20250622 20:11:33.665266 23368 raft_consensus.cc:357] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.665311 23370 raft_consensus.cc:357] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.666057 23368 raft_consensus.cc:383] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:33.666056 23370 raft_consensus.cc:383] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:33.666324 23368 raft_consensus.cc:738] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab55f036561a4c6991c7edcabc7b8c19, State: Initialized, Role: FOLLOWER
I20250622 20:11:33.666324 23370 raft_consensus.cc:738] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a91ef9bddc24c4c994624aa9ec1f34c, State: Initialized, Role: FOLLOWER
I20250622 20:11:33.667084 23370 consensus_queue.cc:260] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.667090 23368 consensus_queue.cc:260] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.672402 23370 ts_tablet_manager.cc:1428] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Time spent starting tablet: real 0.032s user 0.017s sys 0.013s
I20250622 20:11:33.676870 23368 ts_tablet_manager.cc:1428] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Time spent starting tablet: real 0.040s user 0.028s sys 0.009s
W20250622 20:11:33.689028 22949 tablet.cc:2378] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:11:33.769054 23082 tablet.cc:2378] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:11:33.829005 23215 tablet.cc:2378] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:11:33.854316 23375 raft_consensus.cc:491] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:11:33.854825 23375 raft_consensus.cc:513] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.857584 23375 leader_election.cc:290] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1facec447a334a9f9955a0a8212f5d3c (127.17.164.131:34681), ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699)
I20250622 20:11:33.869256 23036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c914b2b829447b08af5f775a7b84f5c" candidate_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1facec447a334a9f9955a0a8212f5d3c" is_pre_election: true
I20250622 20:11:33.870129 23036 raft_consensus.cc:2466] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a91ef9bddc24c4c994624aa9ec1f34c in term 0.
I20250622 20:11:33.871194 23102 leader_election.cc:304] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1facec447a334a9f9955a0a8212f5d3c, 3a91ef9bddc24c4c994624aa9ec1f34c; no voters:
I20250622 20:11:33.871152 22903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c914b2b829447b08af5f775a7b84f5c" candidate_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab55f036561a4c6991c7edcabc7b8c19" is_pre_election: true
I20250622 20:11:33.871816 23375 raft_consensus.cc:2802] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:11:33.872051 23375 raft_consensus.cc:491] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:11:33.871757 22903 raft_consensus.cc:2466] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a91ef9bddc24c4c994624aa9ec1f34c in term 0.
I20250622 20:11:33.872596 23375 raft_consensus.cc:3058] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:33.877132 23375 raft_consensus.cc:513] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.879238 23375 leader_election.cc:290] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [CANDIDATE]: Term 1 election: Requested vote from peers 1facec447a334a9f9955a0a8212f5d3c (127.17.164.131:34681), ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699)
I20250622 20:11:33.879829 23036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c914b2b829447b08af5f775a7b84f5c" candidate_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1facec447a334a9f9955a0a8212f5d3c"
I20250622 20:11:33.880062 22903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9c914b2b829447b08af5f775a7b84f5c" candidate_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab55f036561a4c6991c7edcabc7b8c19"
I20250622 20:11:33.880218 23036 raft_consensus.cc:3058] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:33.880509 22903 raft_consensus.cc:3058] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:33.884979 23036 raft_consensus.cc:2466] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a91ef9bddc24c4c994624aa9ec1f34c in term 1.
I20250622 20:11:33.885223 22903 raft_consensus.cc:2466] T 9c914b2b829447b08af5f775a7b84f5c P ab55f036561a4c6991c7edcabc7b8c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a91ef9bddc24c4c994624aa9ec1f34c in term 1.
I20250622 20:11:33.885746 23102 leader_election.cc:304] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1facec447a334a9f9955a0a8212f5d3c, 3a91ef9bddc24c4c994624aa9ec1f34c; no voters:
I20250622 20:11:33.886261 23375 raft_consensus.cc:2802] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:33.887987 23375 raft_consensus.cc:695] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [term 1 LEADER]: Becoming Leader. State: Replica: 3a91ef9bddc24c4c994624aa9ec1f34c, State: Running, Role: LEADER
I20250622 20:11:33.888659 23375 consensus_queue.cc:237] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } }
I20250622 20:11:33.900430 22625 catalog_manager.cc:5582] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a91ef9bddc24c4c994624aa9ec1f34c (127.17.164.132). New cstate: current_term: 1 leader_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a91ef9bddc24c4c994624aa9ec1f34c" member_type: VOTER last_known_addr { host: "127.17.164.132" port: 39219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab55f036561a4c6991c7edcabc7b8c19" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 42699 } health_report { overall_health: UNKNOWN } } }
I20250622 20:11:33.959623 18066 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250622 20:11:33.963186 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ab55f036561a4c6991c7edcabc7b8c19 to finish bootstrapping
I20250622 20:11:33.976277 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1facec447a334a9f9955a0a8212f5d3c to finish bootstrapping
I20250622 20:11:33.986796 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3a91ef9bddc24c4c994624aa9ec1f34c to finish bootstrapping
I20250622 20:11:33.997870 18066 test_util.cc:276] Using random seed: -1280884454
I20250622 20:11:34.021325 18066 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250622 20:11:34.022136 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22819
W20250622 20:11:34.055521 23102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.130:42699: connect: Connection refused (error 111)
I20250622 20:11:34.056505 23036 raft_consensus.cc:1273] T 9c914b2b829447b08af5f775a7b84f5c P 1facec447a334a9f9955a0a8212f5d3c [term 1 FOLLOWER]: Refusing update from remote peer 3a91ef9bddc24c4c994624aa9ec1f34c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:11:34.059228 23375 consensus_queue.cc:1035] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1facec447a334a9f9955a0a8212f5d3c" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 34681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 20:11:34.061791 23102 consensus_peers.cc:487] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c -> Peer ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699): Couldn't send request to peer ab55f036561a4c6991c7edcabc7b8c19. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.130:42699: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 20:11:34.076787 23390 mvcc.cc:204] Tried to move back new op lower bound from 7170552193234309120 to 7170552192574287872. Current Snapshot: MvccSnapshot[applied={T|T < 7170552193234309120}]
I20250622 20:11:34.082763 23393 mvcc.cc:204] Tried to move back new op lower bound from 7170552193234309120 to 7170552192574287872. Current Snapshot: MvccSnapshot[applied={T|T < 7170552193234309120}]
I20250622 20:11:34.446632 23348 heartbeater.cc:499] Master 127.17.164.190:36471 was elected leader, sending a full tablet report...
W20250622 20:11:36.149776 23102 consensus_peers.cc:487] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c -> Peer ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699): Couldn't send request to peer ab55f036561a4c6991c7edcabc7b8c19. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.130:42699: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 20:11:36.649729 23016 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:11:36.651394 23283 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:11:36.651791 23149 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:11:36.677405 22750 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 20:11:36.740970 23210 debug-util.cc:398] Leaking SignalData structure 0x7b08000c9000 after lost signal to thread 23086
I20250622 20:11:38.559309 23283 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:11:38.572072 23149 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:11:38.593533 22750 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:11:38.622617 23016 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 20:11:38.915937 23102 consensus_peers.cc:487] T 9c914b2b829447b08af5f775a7b84f5c P 3a91ef9bddc24c4c994624aa9ec1f34c -> Peer ab55f036561a4c6991c7edcabc7b8c19 (127.17.164.130:42699): Couldn't send request to peer ab55f036561a4c6991c7edcabc7b8c19. Status: Network error: Client connection negotiation failed: client connection to 127.17.164.130:42699: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250622 20:11:39.261561 23102 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.130:42699: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250622 20:11:41.184394 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22686
I20250622 20:11:41.207443 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22952
I20250622 20:11:41.239733 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23085
I20250622 20:11:41.276469 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23218
I20250622 20:11:41.301079 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 22593
2025-06-22T20:11:41Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19231 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19231 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250622 20:11:41.362587 18066 test_util.cc:276] Using random seed: -1273519747
I20250622 20:11:41.366583 18066 ts_itest-base.cc:115] Starting cluster with:
I20250622 20:11:41.366740 18066 ts_itest-base.cc:116] --------------
I20250622 20:11:41.366909 18066 ts_itest-base.cc:117] 1 tablet servers
I20250622 20:11:41.367048 18066 ts_itest-base.cc:118] 1 replicas per TS
I20250622 20:11:41.367199 18066 ts_itest-base.cc:119] --------------
2025-06-22T20:11:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:11:41Z Disabled control of system clock
I20250622 20:11:41.409631 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:38633
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:38633 with env {}
W20250622 20:11:41.695962 23497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:41.696528 23497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:41.696991 23497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:41.726843 23497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:11:41.727191 23497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:41.727448 23497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:11:41.727690 23497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:11:41.761129 23497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:38633
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:41.762462 23497 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:41.764065 23497 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:41.777999 23503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:41.777992 23504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:41.777976 23506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:42.903313 23505 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:42.903422 23497 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:42.906942 23497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:42.909411 23497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:42.910722 23497 hybrid_clock.cc:648] HybridClock initialized: now 1750623102910693 us; error 80 us; skew 500 ppm
I20250622 20:11:42.911558 23497 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:42.920488 23497 webserver.cc:469] Webserver started at http://127.17.164.190:37905/ using document root <none> and password file <none>
I20250622 20:11:42.921439 23497 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:42.921639 23497 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:42.922122 23497 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:42.928015 23497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b63e8fb799164723bd047c1da969bc24"
format_stamp: "Formatted at 2025-06-22 20:11:42 on dist-test-slave-9mkf"
I20250622 20:11:42.929134 23497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b63e8fb799164723bd047c1da969bc24"
format_stamp: "Formatted at 2025-06-22 20:11:42 on dist-test-slave-9mkf"
I20250622 20:11:42.936257 23497 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250622 20:11:42.941429 23513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:42.942498 23497 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250622 20:11:42.942781 23497 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b63e8fb799164723bd047c1da969bc24"
format_stamp: "Formatted at 2025-06-22 20:11:42 on dist-test-slave-9mkf"
I20250622 20:11:42.943084 23497 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:42.988271 23497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:42.989739 23497 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:42.990180 23497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:43.057863 23497 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:38633
I20250622 20:11:43.057940 23564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:38633 every 8 connection(s)
I20250622 20:11:43.060406 23497 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250622 20:11:43.065476 23565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:43.067420 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23497
I20250622 20:11:43.067816 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250622 20:11:43.087642 23565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24: Bootstrap starting.
I20250622 20:11:43.094009 23565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:43.095798 23565 log.cc:826] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:43.099738 23565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24: No bootstrap required, opened a new log
I20250622 20:11:43.116317 23565 raft_consensus.cc:357] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } }
I20250622 20:11:43.116976 23565 raft_consensus.cc:383] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:43.117201 23565 raft_consensus.cc:738] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63e8fb799164723bd047c1da969bc24, State: Initialized, Role: FOLLOWER
I20250622 20:11:43.117911 23565 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } }
I20250622 20:11:43.118402 23565 raft_consensus.cc:397] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:11:43.118652 23565 raft_consensus.cc:491] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:11:43.118932 23565 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:43.122938 23565 raft_consensus.cc:513] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } }
I20250622 20:11:43.123543 23565 leader_election.cc:304] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b63e8fb799164723bd047c1da969bc24; no voters:
I20250622 20:11:43.125109 23565 leader_election.cc:290] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:11:43.125767 23570 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:43.128585 23570 raft_consensus.cc:695] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [term 1 LEADER]: Becoming Leader. State: Replica: b63e8fb799164723bd047c1da969bc24, State: Running, Role: LEADER
I20250622 20:11:43.129328 23565 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:11:43.129343 23570 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } }
I20250622 20:11:43.139114 23571 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b63e8fb799164723bd047c1da969bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } } }
I20250622 20:11:43.139600 23571 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:43.140079 23572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63e8fb799164723bd047c1da969bc24. Latest consensus state: current_term: 1 leader_uuid: "b63e8fb799164723bd047c1da969bc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63e8fb799164723bd047c1da969bc24" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 38633 } } }
I20250622 20:11:43.140679 23572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:43.143203 23577 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:11:43.154129 23577 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:11:43.170214 23577 catalog_manager.cc:1349] Generated new cluster ID: e41525ecc31045dbbcb2aeeabbf8a6a3
I20250622 20:11:43.170521 23577 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:11:43.218961 23577 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:11:43.220315 23577 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:11:43.244088 23577 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b63e8fb799164723bd047c1da969bc24: Generated new TSK 0
I20250622 20:11:43.245150 23577 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:11:43.265771 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:38633
--builtin_ntp_servers=127.17.164.148:39983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250622 20:11:43.565649 23589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:43.566151 23589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:43.566660 23589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:43.599380 23589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:43.600231 23589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:11:43.636744 23589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:39983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:38633
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:43.638082 23589 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:43.639580 23589 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:43.656309 23596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:43.656695 23597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:43.658619 23599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:43.658816 23589 server_base.cc:1048] running on GCE node
I20250622 20:11:44.816301 23589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:44.821357 23589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:44.822767 23589 hybrid_clock.cc:648] HybridClock initialized: now 1750623104822714 us; error 72 us; skew 500 ppm
I20250622 20:11:44.823546 23589 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:44.830325 23589 webserver.cc:469] Webserver started at http://127.17.164.129:42027/ using document root <none> and password file <none>
I20250622 20:11:44.831214 23589 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:44.831436 23589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:44.831903 23589 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:44.836184 23589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6e70cae9cc1547a18a8713f611950400"
format_stamp: "Formatted at 2025-06-22 20:11:44 on dist-test-slave-9mkf"
I20250622 20:11:44.837248 23589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6e70cae9cc1547a18a8713f611950400"
format_stamp: "Formatted at 2025-06-22 20:11:44 on dist-test-slave-9mkf"
I20250622 20:11:44.844363 23589 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250622 20:11:44.850044 23606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:44.851040 23589 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250622 20:11:44.851336 23589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6e70cae9cc1547a18a8713f611950400"
format_stamp: "Formatted at 2025-06-22 20:11:44 on dist-test-slave-9mkf"
I20250622 20:11:44.851642 23589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250622 20:11:44.861016 23561 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 23498
W20250622 20:11:44.862128 23561 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 23564
I20250622 20:11:44.927103 23589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:44.928617 23589 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:44.929068 23589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:44.931758 23589 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:44.936421 23589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:44.936625 23589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:44.936909 23589 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:44.937053 23589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:45.095012 23589 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:42043
I20250622 20:11:45.095114 23718 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:42043 every 8 connection(s)
I20250622 20:11:45.097580 23589 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250622 20:11:45.102028 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23589
I20250622 20:11:45.102921 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750622963291160-18066-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250622 20:11:45.125555 23719 heartbeater.cc:344] Connected to a master server at 127.17.164.190:38633
I20250622 20:11:45.126078 23719 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:45.127409 23719 heartbeater.cc:507] Master 127.17.164.190:38633 requested a full tablet report, sending...
I20250622 20:11:45.130352 23529 ts_manager.cc:194] Registered new tserver with Master: 6e70cae9cc1547a18a8713f611950400 (127.17.164.129:42043)
I20250622 20:11:45.133258 23529 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:44645
I20250622 20:11:45.139549 18066 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 20:11:45.176410 23529 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52808:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250622 20:11:45.251747 23654 tablet_service.cc:1468] Processing CreateTablet for tablet 87c804ff90be43c7b74d5e3993406470 (DEFAULT_TABLE table=TestTable [id=5c6951ecc908432882ad4a156f8f869a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:45.253690 23654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87c804ff90be43c7b74d5e3993406470. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:45.283351 23734 tablet_bootstrap.cc:492] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: Bootstrap starting.
I20250622 20:11:45.288851 23734 tablet_bootstrap.cc:654] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:45.290640 23734 log.cc:826] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:45.295511 23734 tablet_bootstrap.cc:492] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: No bootstrap required, opened a new log
I20250622 20:11:45.295920 23734 ts_tablet_manager.cc:1397] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20250622 20:11:45.312856 23734 raft_consensus.cc:357] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e70cae9cc1547a18a8713f611950400" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 42043 } }
I20250622 20:11:45.313397 23734 raft_consensus.cc:383] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:45.313628 23734 raft_consensus.cc:738] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e70cae9cc1547a18a8713f611950400, State: Initialized, Role: FOLLOWER
I20250622 20:11:45.314325 23734 consensus_queue.cc:260] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e70cae9cc1547a18a8713f611950400" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 42043 } }
I20250622 20:11:45.314812 23734 raft_consensus.cc:397] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:11:45.315076 23734 raft_consensus.cc:491] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:11:45.315346 23734 raft_consensus.cc:3058] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:45.319320 23734 raft_consensus.cc:513] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e70cae9cc1547a18a8713f611950400" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 42043 } }
I20250622 20:11:45.320000 23734 leader_election.cc:304] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e70cae9cc1547a18a8713f611950400; no voters:
I20250622 20:11:45.322223 23734 leader_election.cc:290] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:11:45.322639 23736 raft_consensus.cc:2802] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:45.325150 23736 raft_consensus.cc:695] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [term 1 LEADER]: Becoming Leader. State: Replica: 6e70cae9cc1547a18a8713f611950400, State: Running, Role: LEADER
I20250622 20:11:45.326226 23719 heartbeater.cc:499] Master 127.17.164.190:38633 was elected leader, sending a full tablet report...
I20250622 20:11:45.326283 23736 consensus_queue.cc:237] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e70cae9cc1547a18a8713f611950400" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 42043 } }
I20250622 20:11:45.329509 23734 ts_tablet_manager.cc:1428] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400: Time spent starting tablet: real 0.033s user 0.016s sys 0.019s
I20250622 20:11:45.339728 23529 catalog_manager.cc:5582] T 87c804ff90be43c7b74d5e3993406470 P 6e70cae9cc1547a18a8713f611950400 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e70cae9cc1547a18a8713f611950400 (127.17.164.129). New cstate: current_term: 1 leader_uuid: "6e70cae9cc1547a18a8713f611950400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e70cae9cc1547a18a8713f611950400" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 42043 } health_report { overall_health: HEALTHY } } }
I20250622 20:11:45.382911 18066 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250622 20:11:45.386758 18066 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6e70cae9cc1547a18a8713f611950400 to finish bootstrapping
I20250622 20:11:48.072881 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23589
I20250622 20:11:48.098145 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23497
2025-06-22T20:11:48Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6792 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6792 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250622 20:11:48.155567 18066 test_util.cc:276] Using random seed: -1266726762
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-22T20:11:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T20:11:48Z Disabled control of system clock
I20250622 20:11:48.204604 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:44657
--webserver_interface=127.17.164.190
--webserver_port=0
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:44657 with env {}
W20250622 20:11:48.488169 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:48.488673 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:48.489070 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:48.518317 23762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:11:48.518587 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:48.518783 23762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:11:48.518981 23762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:11:48.552284 23762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:44657
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:48.553558 23762 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:48.555071 23762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:48.569792 23769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:48.569834 23771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:48.570987 23768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:49.692278 23770 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:49.692346 23762 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:49.696026 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:49.699003 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:49.700414 23762 hybrid_clock.cc:648] HybridClock initialized: now 1750623109700364 us; error 68 us; skew 500 ppm
I20250622 20:11:49.701179 23762 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:49.712514 23762 webserver.cc:469] Webserver started at http://127.17.164.190:33899/ using document root <none> and password file <none>
I20250622 20:11:49.713476 23762 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:49.713680 23762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:49.714079 23762 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:49.718513 23762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/instance:
uuid: "a2eb1860c4174fc78491bde88807f8a9"
format_stamp: "Formatted at 2025-06-22 20:11:49 on dist-test-slave-9mkf"
I20250622 20:11:49.719513 23762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal/instance:
uuid: "a2eb1860c4174fc78491bde88807f8a9"
format_stamp: "Formatted at 2025-06-22 20:11:49 on dist-test-slave-9mkf"
I20250622 20:11:49.726188 23762 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250622 20:11:49.731104 23778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:49.732024 23762 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 20:11:49.732318 23762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
uuid: "a2eb1860c4174fc78491bde88807f8a9"
format_stamp: "Formatted at 2025-06-22 20:11:49 on dist-test-slave-9mkf"
I20250622 20:11:49.732617 23762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:49.783615 23762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:49.784977 23762 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:49.785385 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:49.852993 23762 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:44657
I20250622 20:11:49.853073 23829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:44657 every 8 connection(s)
I20250622 20:11:49.855553 23762 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
I20250622 20:11:49.858644 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23762
I20250622 20:11:49.859269 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal/instance
I20250622 20:11:49.860225 23830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:49.885426 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9: Bootstrap starting.
I20250622 20:11:49.890364 23830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:49.892051 23830 log.cc:826] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:49.896034 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9: No bootstrap required, opened a new log
I20250622 20:11:49.912580 23830 raft_consensus.cc:357] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:11:49.913285 23830 raft_consensus.cc:383] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:49.913551 23830 raft_consensus.cc:738] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2eb1860c4174fc78491bde88807f8a9, State: Initialized, Role: FOLLOWER
I20250622 20:11:49.914306 23830 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:11:49.914755 23830 raft_consensus.cc:397] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:11:49.915002 23830 raft_consensus.cc:491] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:11:49.915241 23830 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:49.918852 23830 raft_consensus.cc:513] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:11:49.919451 23830 leader_election.cc:304] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2eb1860c4174fc78491bde88807f8a9; no voters:
I20250622 20:11:49.921006 23830 leader_election.cc:290] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:11:49.921725 23835 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:49.924443 23835 raft_consensus.cc:695] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [term 1 LEADER]: Becoming Leader. State: Replica: a2eb1860c4174fc78491bde88807f8a9, State: Running, Role: LEADER
I20250622 20:11:49.924983 23830 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:11:49.925159 23835 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:11:49.936100 23836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2eb1860c4174fc78491bde88807f8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } } }
I20250622 20:11:49.936923 23836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:49.937500 23837 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2eb1860c4174fc78491bde88807f8a9. Latest consensus state: current_term: 1 leader_uuid: "a2eb1860c4174fc78491bde88807f8a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2eb1860c4174fc78491bde88807f8a9" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } } }
I20250622 20:11:49.938292 23837 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9 [sys.catalog]: This master's current role is: LEADER
I20250622 20:11:49.939839 23845 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:11:49.951531 23845 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:11:49.966071 23845 catalog_manager.cc:1349] Generated new cluster ID: a865767ea7964be7b5ae5db15821855c
I20250622 20:11:49.966382 23845 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:11:49.993337 23845 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:11:49.994720 23845 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:11:50.016244 23845 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a2eb1860c4174fc78491bde88807f8a9: Generated new TSK 0
I20250622 20:11:50.017261 23845 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 20:11:50.032320 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:0
--local_ip_for_outbound_sockets=127.17.164.129
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250622 20:11:50.326392 23854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:50.326864 23854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:50.327359 23854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:50.356673 23854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:50.357568 23854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:11:50.391430 23854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:50.392692 23854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:50.394280 23854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:50.410825 23861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:50.410873 23860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:50.411650 23863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:51.901260 23862 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1488 milliseconds
I20250622 20:11:51.901382 23854 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:51.902532 23854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:51.904595 23854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:51.906028 23854 hybrid_clock.cc:648] HybridClock initialized: now 1750623111905989 us; error 54 us; skew 500 ppm
I20250622 20:11:51.906827 23854 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:51.912889 23854 webserver.cc:469] Webserver started at http://127.17.164.129:45623/ using document root <none> and password file <none>
I20250622 20:11:51.913812 23854 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:51.913993 23854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:51.914466 23854 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:51.918772 23854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/instance:
uuid: "34d1635f5f2f402d849ed0bdbf527b2f"
format_stamp: "Formatted at 2025-06-22 20:11:51 on dist-test-slave-9mkf"
I20250622 20:11:51.919795 23854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal/instance:
uuid: "34d1635f5f2f402d849ed0bdbf527b2f"
format_stamp: "Formatted at 2025-06-22 20:11:51 on dist-test-slave-9mkf"
I20250622 20:11:51.926630 23854 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250622 20:11:51.932045 23870 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:51.933084 23854 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 20:11:51.933395 23854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
uuid: "34d1635f5f2f402d849ed0bdbf527b2f"
format_stamp: "Formatted at 2025-06-22 20:11:51 on dist-test-slave-9mkf"
I20250622 20:11:51.933706 23854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:51.999079 23854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:52.000602 23854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:52.001036 23854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:52.003381 23854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:52.007190 23854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:52.007365 23854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:52.007619 23854 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:52.007776 23854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:52.141646 23854 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:41759
I20250622 20:11:52.141753 23982 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:41759 every 8 connection(s)
I20250622 20:11:52.144130 23854 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
I20250622 20:11:52.154098 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23854
I20250622 20:11:52.154582 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal/instance
I20250622 20:11:52.160944 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:0
--local_ip_for_outbound_sockets=127.17.164.130
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 20:11:52.165392 23983 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:11:52.165791 23983 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:52.166800 23983 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:11:52.169293 23795 ts_manager.cc:194] Registered new tserver with Master: 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129:41759)
I20250622 20:11:52.171087 23795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:57821
W20250622 20:11:52.453006 23987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:52.453503 23987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:52.453995 23987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:52.483570 23987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:52.484418 23987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:11:52.517272 23987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:52.518563 23987 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:52.520169 23987 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:52.534703 23993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:53.174347 23983 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
W20250622 20:11:52.535091 23994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:52.537340 23996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:53.659998 23995 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:11:53.660072 23987 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:53.663992 23987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:53.666833 23987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:53.668255 23987 hybrid_clock.cc:648] HybridClock initialized: now 1750623113668221 us; error 59 us; skew 500 ppm
I20250622 20:11:53.669080 23987 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:53.675669 23987 webserver.cc:469] Webserver started at http://127.17.164.130:33829/ using document root <none> and password file <none>
I20250622 20:11:53.676544 23987 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:53.676766 23987 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:53.677206 23987 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:53.681439 23987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/instance:
uuid: "609af6f2666d47d09a79fc14319ce9ff"
format_stamp: "Formatted at 2025-06-22 20:11:53 on dist-test-slave-9mkf"
I20250622 20:11:53.682435 23987 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal/instance:
uuid: "609af6f2666d47d09a79fc14319ce9ff"
format_stamp: "Formatted at 2025-06-22 20:11:53 on dist-test-slave-9mkf"
I20250622 20:11:53.689520 23987 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 20:11:53.694718 24004 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:53.695708 23987 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250622 20:11:53.696019 23987 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
uuid: "609af6f2666d47d09a79fc14319ce9ff"
format_stamp: "Formatted at 2025-06-22 20:11:53 on dist-test-slave-9mkf"
I20250622 20:11:53.696332 23987 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:53.741881 23987 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:53.743294 23987 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:53.743711 23987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:53.746042 23987 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:53.749902 23987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:53.750111 23987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:53.750339 23987 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:53.750494 23987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:53.879899 23987 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:36941
I20250622 20:11:53.880028 24116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:36941 every 8 connection(s)
I20250622 20:11:53.882378 23987 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
I20250622 20:11:53.889356 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 23987
I20250622 20:11:53.889963 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal/instance
I20250622 20:11:53.895864 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:0
--local_ip_for_outbound_sockets=127.17.164.131
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 20:11:53.902194 24117 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:11:53.902592 24117 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:53.903542 24117 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:11:53.905619 23795 ts_manager.cc:194] Registered new tserver with Master: 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941)
I20250622 20:11:53.906762 23795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:51005
W20250622 20:11:54.184062 24121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:11:54.184552 24121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:11:54.185027 24121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:11:54.213891 24121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:11:54.214695 24121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:11:54.246568 24121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=0
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:11:54.247790 24121 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:11:54.249362 24121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:11:54.262889 24128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:54.909600 24117 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
W20250622 20:11:54.263965 24127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:11:55.368127 24129 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250622 20:11:55.369453 24130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:11:55.369469 24121 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:11:55.372241 24121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:11:55.374907 24121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:11:55.376319 24121 hybrid_clock.cc:648] HybridClock initialized: now 1750623115376264 us; error 65 us; skew 500 ppm
I20250622 20:11:55.377152 24121 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:11:55.386839 24121 webserver.cc:469] Webserver started at http://127.17.164.131:34361/ using document root <none> and password file <none>
I20250622 20:11:55.387693 24121 fs_manager.cc:362] Metadata directory not provided
I20250622 20:11:55.387881 24121 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:11:55.388362 24121 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:11:55.393209 24121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/instance:
uuid: "7a56eca9c21c4b37822574bf53c547a3"
format_stamp: "Formatted at 2025-06-22 20:11:55 on dist-test-slave-9mkf"
I20250622 20:11:55.394418 24121 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal/instance:
uuid: "7a56eca9c21c4b37822574bf53c547a3"
format_stamp: "Formatted at 2025-06-22 20:11:55 on dist-test-slave-9mkf"
I20250622 20:11:55.401188 24121 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250622 20:11:55.406577 24137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:55.407588 24121 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250622 20:11:55.407934 24121 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
uuid: "7a56eca9c21c4b37822574bf53c547a3"
format_stamp: "Formatted at 2025-06-22 20:11:55 on dist-test-slave-9mkf"
I20250622 20:11:55.408565 24121 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:11:55.476274 24121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:11:55.477718 24121 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:11:55.478148 24121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:11:55.480454 24121 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:11:55.484342 24121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 20:11:55.484524 24121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:55.484736 24121 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 20:11:55.484865 24121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 20:11:55.615852 24121 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:33993
I20250622 20:11:55.615944 24249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:33993 every 8 connection(s)
I20250622 20:11:55.618304 24121 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
I20250622 20:11:55.625229 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 24121
I20250622 20:11:55.625646 18066 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal/instance
I20250622 20:11:55.638588 24250 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:11:55.638978 24250 heartbeater.cc:461] Registering TS with master...
I20250622 20:11:55.639951 24250 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:11:55.641829 23795 ts_manager.cc:194] Registered new tserver with Master: 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:11:55.643384 23795 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:40417
I20250622 20:11:55.645399 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:11:55.675908 18066 test_util.cc:276] Using random seed: -1259206418
I20250622 20:11:55.711993 23795 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46866:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250622 20:11:55.714259 23795 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:11:55.774906 23918 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa7a0acc08741378194be66d49845bd (DEFAULT_TABLE table=pre_rebuild [id=29f70792bd704efb95db133a38052d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:55.774904 24052 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa7a0acc08741378194be66d49845bd (DEFAULT_TABLE table=pre_rebuild [id=29f70792bd704efb95db133a38052d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:55.776297 24185 tablet_service.cc:1468] Processing CreateTablet for tablet 8aa7a0acc08741378194be66d49845bd (DEFAULT_TABLE table=pre_rebuild [id=29f70792bd704efb95db133a38052d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:11:55.776779 24052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa7a0acc08741378194be66d49845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:55.776780 23918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa7a0acc08741378194be66d49845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:55.777778 24185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aa7a0acc08741378194be66d49845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:11:55.799635 24274 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Bootstrap starting.
I20250622 20:11:55.803530 24276 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Bootstrap starting.
I20250622 20:11:55.806138 24275 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Bootstrap starting.
I20250622 20:11:55.806803 24274 tablet_bootstrap.cc:654] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:55.809223 24274 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:55.812363 24275 tablet_bootstrap.cc:654] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:55.813088 24276 tablet_bootstrap.cc:654] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Neither blocks nor log segments found. Creating new log.
I20250622 20:11:55.814170 24275 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:55.814749 24274 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: No bootstrap required, opened a new log
I20250622 20:11:55.815299 24274 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250622 20:11:55.815289 24276 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Log is configured to *not* fsync() on all Append() calls
I20250622 20:11:55.818774 24275 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: No bootstrap required, opened a new log
I20250622 20:11:55.819204 24275 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.007s
I20250622 20:11:55.822425 24276 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: No bootstrap required, opened a new log
I20250622 20:11:55.822901 24276 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.008s
I20250622 20:11:55.837478 24275 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.838155 24275 raft_consensus.cc:383] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:55.838414 24275 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1635f5f2f402d849ed0bdbf527b2f, State: Initialized, Role: FOLLOWER
I20250622 20:11:55.839154 24275 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.843796 24274 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.844633 24274 raft_consensus.cc:383] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:55.844970 24274 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a56eca9c21c4b37822574bf53c547a3, State: Initialized, Role: FOLLOWER
I20250622 20:11:55.845808 24275 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent starting tablet: real 0.026s user 0.021s sys 0.003s
I20250622 20:11:55.845817 24274 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.847669 24276 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.848730 24276 raft_consensus.cc:383] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:11:55.849022 24250 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
I20250622 20:11:55.849035 24276 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609af6f2666d47d09a79fc14319ce9ff, State: Initialized, Role: FOLLOWER
I20250622 20:11:55.849942 24274 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Time spent starting tablet: real 0.034s user 0.026s sys 0.007s
I20250622 20:11:55.850190 24276 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:55.860038 24276 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Time spent starting tablet: real 0.037s user 0.027s sys 0.012s
W20250622 20:11:55.872710 24251 tablet.cc:2378] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:11:55.888640 24118 tablet.cc:2378] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:11:55.902689 23984 tablet.cc:2378] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:11:56.020639 24281 raft_consensus.cc:491] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:11:56.021078 24281 raft_consensus.cc:513] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:56.023315 24281 leader_election.cc:290] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129:41759), 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941)
I20250622 20:11:56.034683 23938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "7a56eca9c21c4b37822574bf53c547a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" is_pre_election: true
I20250622 20:11:56.035626 23938 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a56eca9c21c4b37822574bf53c547a3 in term 0.
I20250622 20:11:56.035583 24072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "7a56eca9c21c4b37822574bf53c547a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff" is_pre_election: true
I20250622 20:11:56.036409 24072 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a56eca9c21c4b37822574bf53c547a3 in term 0.
I20250622 20:11:56.037003 24140 leader_election.cc:304] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f, 7a56eca9c21c4b37822574bf53c547a3; no voters:
I20250622 20:11:56.037829 24281 raft_consensus.cc:2802] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:11:56.038077 24281 raft_consensus.cc:491] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:11:56.038360 24281 raft_consensus.cc:3058] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:56.042688 24281 raft_consensus.cc:513] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:56.043942 24281 leader_election.cc:290] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [CANDIDATE]: Term 1 election: Requested vote from peers 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129:41759), 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941)
I20250622 20:11:56.044668 23938 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "7a56eca9c21c4b37822574bf53c547a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34d1635f5f2f402d849ed0bdbf527b2f"
I20250622 20:11:56.044782 24072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "7a56eca9c21c4b37822574bf53c547a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff"
I20250622 20:11:56.045149 23938 raft_consensus.cc:3058] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:56.045238 24072 raft_consensus.cc:3058] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:11:56.049412 23938 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a56eca9c21c4b37822574bf53c547a3 in term 1.
I20250622 20:11:56.049389 24072 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a56eca9c21c4b37822574bf53c547a3 in term 1.
I20250622 20:11:56.050321 24139 leader_election.cc:304] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 609af6f2666d47d09a79fc14319ce9ff, 7a56eca9c21c4b37822574bf53c547a3; no voters:
I20250622 20:11:56.050940 24281 raft_consensus.cc:2802] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:11:56.052390 24281 raft_consensus.cc:695] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a56eca9c21c4b37822574bf53c547a3, State: Running, Role: LEADER
I20250622 20:11:56.053160 24281 consensus_queue.cc:237] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:11:56.064013 23794 catalog_manager.cc:5582] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131). New cstate: current_term: 1 leader_uuid: "7a56eca9c21c4b37822574bf53c547a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } health_report { overall_health: HEALTHY } } }
I20250622 20:11:56.300730 23938 raft_consensus.cc:1273] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Refusing update from remote peer 7a56eca9c21c4b37822574bf53c547a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:11:56.300738 24072 raft_consensus.cc:1273] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Refusing update from remote peer 7a56eca9c21c4b37822574bf53c547a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:11:56.302430 24281 consensus_queue.cc:1035] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 20:11:56.303489 24284 consensus_queue.cc:1035] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:11:56.328370 24293 mvcc.cc:204] Tried to move back new op lower bound from 7170552284350222336 to 7170552283360514048. Current Snapshot: MvccSnapshot[applied={T|T < 7170552284350222336}]
I20250622 20:11:56.370461 24294 mvcc.cc:204] Tried to move back new op lower bound from 7170552284370608128 to 7170552283360514048. Current Snapshot: MvccSnapshot[applied={T|T < 7170552284370608128}]
I20250622 20:11:56.382287 24295 mvcc.cc:204] Tried to move back new op lower bound from 7170552284350222336 to 7170552283360514048. Current Snapshot: MvccSnapshot[applied={T|T < 7170552284350222336}]
I20250622 20:12:01.385215 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23762
W20250622 20:12:01.415966 24138 connection.cc:537] client connection to 127.17.164.190:44657 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 20:12:01.416919 24250 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:44657 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 20:12:01.466935 23983 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:44657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:44657: connect: Connection refused (error 111)
W20250622 20:12:01.726915 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:12:01.727536 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:12:01.758651 24332 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250622 20:12:02.399045 24117 heartbeater.cc:646] Failed to heartbeat to 127.17.164.190:44657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.164.190:44657: connect: Connection refused (error 111)
W20250622 20:12:03.209995 24332 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.408s user 0.545s sys 0.860s
W20250622 20:12:03.210413 24332 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.409s user 0.546s sys 0.861s
I20250622 20:12:03.345328 24332 minidump.cc:252] Setting minidump size limit to 20M
I20250622 20:12:03.347812 24332 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:12:03.349212 24332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:12:03.362259 24363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:03.363363 24364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:03.484963 24332 server_base.cc:1048] running on GCE node
W20250622 20:12:03.484902 24366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:03.486379 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250622 20:12:03.486905 24332 hybrid_clock.cc:648] HybridClock initialized: now 1750623123486875 us; error 153584 us; skew 500 ppm
I20250622 20:12:03.487870 24332 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:12:03.493211 24332 webserver.cc:469] Webserver started at http://0.0.0.0:32937/ using document root <none> and password file <none>
I20250622 20:12:03.494261 24332 fs_manager.cc:362] Metadata directory not provided
I20250622 20:12:03.494558 24332 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:12:03.495090 24332 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 20:12:03.501060 24332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/instance:
uuid: "088253faf14e40618a2efb47083b3156"
format_stamp: "Formatted at 2025-06-22 20:12:03 on dist-test-slave-9mkf"
I20250622 20:12:03.502460 24332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal/instance:
uuid: "088253faf14e40618a2efb47083b3156"
format_stamp: "Formatted at 2025-06-22 20:12:03 on dist-test-slave-9mkf"
I20250622 20:12:03.509716 24332 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250622 20:12:03.515633 24374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:12:03.516527 24332 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250622 20:12:03.516896 24332 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
uuid: "088253faf14e40618a2efb47083b3156"
format_stamp: "Formatted at 2025-06-22 20:12:03 on dist-test-slave-9mkf"
I20250622 20:12:03.517257 24332 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:12:03.635764 24332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:12:03.637717 24332 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:12:03.638329 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:12:03.644157 24332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:12:03.661669 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Bootstrap starting.
I20250622 20:12:03.666226 24332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Neither blocks nor log segments found. Creating new log.
I20250622 20:12:03.667866 24332 log.cc:826] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Log is configured to *not* fsync() on all Append() calls
I20250622 20:12:03.672205 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: No bootstrap required, opened a new log
I20250622 20:12:03.689678 24332 raft_consensus.cc:357] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER }
I20250622 20:12:03.690125 24332 raft_consensus.cc:383] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:12:03.690357 24332 raft_consensus.cc:738] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088253faf14e40618a2efb47083b3156, State: Initialized, Role: FOLLOWER
I20250622 20:12:03.690968 24332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER }
I20250622 20:12:03.691388 24332 raft_consensus.cc:397] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:12:03.691612 24332 raft_consensus.cc:491] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:12:03.691864 24332 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:12:03.695684 24332 raft_consensus.cc:513] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER }
I20250622 20:12:03.696280 24332 leader_election.cc:304] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088253faf14e40618a2efb47083b3156; no voters:
I20250622 20:12:03.698391 24332 leader_election.cc:290] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 20:12:03.698617 24382 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:12:03.702322 24382 raft_consensus.cc:695] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 LEADER]: Becoming Leader. State: Replica: 088253faf14e40618a2efb47083b3156, State: Running, Role: LEADER
I20250622 20:12:03.703127 24382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER }
I20250622 20:12:03.708729 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088253faf14e40618a2efb47083b3156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER } }
I20250622 20:12:03.709070 24384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088253faf14e40618a2efb47083b3156. Latest consensus state: current_term: 1 leader_uuid: "088253faf14e40618a2efb47083b3156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER } }
I20250622 20:12:03.709400 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: This master's current role is: LEADER
I20250622 20:12:03.709573 24384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: This master's current role is: LEADER
I20250622 20:12:03.723966 24332 tablet_replica.cc:331] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: stopping tablet replica
I20250622 20:12:03.724638 24332 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 LEADER]: Raft consensus shutting down.
I20250622 20:12:03.725157 24332 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:12:03.727414 24332 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250622 20:12:03.727777 24332 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250622 20:12:03.753082 24332 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250622 20:12:04.782691 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23854
I20250622 20:12:04.815322 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 23987
I20250622 20:12:04.849661 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 24121
I20250622 20:12:04.890740 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:44657
--webserver_interface=127.17.164.190
--webserver_port=33899
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.164.190:44657 with env {}
W20250622 20:12:05.178261 24392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:12:05.178848 24392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:12:05.179278 24392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:12:05.213793 24392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 20:12:05.214114 24392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:12:05.214355 24392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 20:12:05.214581 24392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 20:12:05.248876 24392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.164.190:44657
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.164.190:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.164.190
--webserver_port=33899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:12:05.250181 24392 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:12:05.251735 24392 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:12:05.266247 24398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:05.266486 24401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:05.266513 24399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:05.267637 24392 server_base.cc:1048] running on GCE node
I20250622 20:12:06.413576 24392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:12:06.416123 24392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:12:06.417462 24392 hybrid_clock.cc:648] HybridClock initialized: now 1750623126417429 us; error 50 us; skew 500 ppm
I20250622 20:12:06.418234 24392 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:12:06.424140 24392 webserver.cc:469] Webserver started at http://127.17.164.190:33899/ using document root <none> and password file <none>
I20250622 20:12:06.425060 24392 fs_manager.cc:362] Metadata directory not provided
I20250622 20:12:06.425238 24392 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:12:06.432533 24392 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250622 20:12:06.436888 24408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:12:06.437852 24392 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 20:12:06.438118 24392 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
uuid: "088253faf14e40618a2efb47083b3156"
format_stamp: "Formatted at 2025-06-22 20:12:03 on dist-test-slave-9mkf"
I20250622 20:12:06.439883 24392 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:12:06.495170 24392 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:12:06.496573 24392 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:12:06.496953 24392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:12:06.564082 24392 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.190:44657
I20250622 20:12:06.564190 24459 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.190:44657 every 8 connection(s)
I20250622 20:12:06.566789 24392 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb
I20250622 20:12:06.567646 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 24392
I20250622 20:12:06.569151 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.129:41759
--local_ip_for_outbound_sockets=127.17.164.129
--tserver_master_addrs=127.17.164.190:44657
--webserver_port=45623
--webserver_interface=127.17.164.129
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 20:12:06.578276 24460 sys_catalog.cc:263] Verifying existing consensus state
I20250622 20:12:06.589346 24460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Bootstrap starting.
I20250622 20:12:06.598744 24460 log.cc:826] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Log is configured to *not* fsync() on all Append() calls
I20250622 20:12:06.609808 24460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:12:06.610561 24460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Bootstrap complete.
I20250622 20:12:06.630352 24460 raft_consensus.cc:357] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:12:06.630993 24460 raft_consensus.cc:738] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088253faf14e40618a2efb47083b3156, State: Initialized, Role: FOLLOWER
I20250622 20:12:06.631726 24460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:12:06.632201 24460 raft_consensus.cc:397] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 20:12:06.632453 24460 raft_consensus.cc:491] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 20:12:06.632758 24460 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:12:06.636634 24460 raft_consensus.cc:513] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:12:06.637265 24460 leader_election.cc:304] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088253faf14e40618a2efb47083b3156; no voters:
I20250622 20:12:06.639344 24460 leader_election.cc:290] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250622 20:12:06.639672 24464 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:12:06.642935 24464 raft_consensus.cc:695] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [term 2 LEADER]: Becoming Leader. State: Replica: 088253faf14e40618a2efb47083b3156, State: Running, Role: LEADER
I20250622 20:12:06.643800 24464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } }
I20250622 20:12:06.644575 24460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: configured and running, proceeding with master startup.
I20250622 20:12:06.654165 24465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "088253faf14e40618a2efb47083b3156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } } }
I20250622 20:12:06.655912 24465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: This master's current role is: LEADER
I20250622 20:12:06.655349 24466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088253faf14e40618a2efb47083b3156. Latest consensus state: current_term: 2 leader_uuid: "088253faf14e40618a2efb47083b3156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088253faf14e40618a2efb47083b3156" member_type: VOTER last_known_addr { host: "127.17.164.190" port: 44657 } } }
I20250622 20:12:06.657771 24466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156 [sys.catalog]: This master's current role is: LEADER
I20250622 20:12:06.665611 24471 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 20:12:06.677398 24471 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=6857e9d3b33d4d11ac2a7c2310eeb8b6]
I20250622 20:12:06.684128 24471 tablet_loader.cc:96] loaded metadata for tablet 8aa7a0acc08741378194be66d49845bd (table pre_rebuild [id=6857e9d3b33d4d11ac2a7c2310eeb8b6])
I20250622 20:12:06.685573 24471 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 20:12:06.713114 24471 catalog_manager.cc:1349] Generated new cluster ID: cb9a7778d7da404384a6c0ed68dae587
I20250622 20:12:06.713508 24471 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 20:12:06.731480 24471 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 20:12:06.732476 24482 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250622 20:12:06.733620 24471 catalog_manager.cc:1506] Loading token signing keys...
I20250622 20:12:06.753597 24471 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Generated new TSK 0
I20250622 20:12:06.754742 24471 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250622 20:12:06.906363 24462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:12:06.906891 24462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:12:06.907398 24462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:12:06.938380 24462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:12:06.939249 24462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.129
I20250622 20:12:06.973789 24462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.129:41759
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.164.129
--webserver_port=45623
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.129
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:12:06.975095 24462 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:12:06.976668 24462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:12:06.993973 24488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:06.995016 24489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:06.994539 24491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:08.185719 24490 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250622 20:12:08.186273 24462 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:12:08.189685 24462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:12:08.192330 24462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:12:08.193821 24462 hybrid_clock.cc:648] HybridClock initialized: now 1750623128193730 us; error 110 us; skew 500 ppm
I20250622 20:12:08.194667 24462 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:12:08.201926 24462 webserver.cc:469] Webserver started at http://127.17.164.129:45623/ using document root <none> and password file <none>
I20250622 20:12:08.202826 24462 fs_manager.cc:362] Metadata directory not provided
I20250622 20:12:08.203092 24462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:12:08.211015 24462 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20250622 20:12:08.215835 24498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:12:08.217011 24462 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 20:12:08.217325 24462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
uuid: "34d1635f5f2f402d849ed0bdbf527b2f"
format_stamp: "Formatted at 2025-06-22 20:11:51 on dist-test-slave-9mkf"
I20250622 20:12:08.219252 24462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:12:08.285243 24462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:12:08.286696 24462 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:12:08.287135 24462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:12:08.289767 24462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:12:08.295981 24505 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:12:08.306761 24462 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:12:08.307013 24462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250622 20:12:08.307284 24462 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:12:08.312026 24462 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:12:08.312217 24462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250622 20:12:08.312665 24505 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Bootstrap starting.
I20250622 20:12:08.507329 24462 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.129:41759
I20250622 20:12:08.507400 24611 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.129:41759 every 8 connection(s)
I20250622 20:12:08.510967 24462 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb
I20250622 20:12:08.513239 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 24462
I20250622 20:12:08.515281 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.130:36941
--local_ip_for_outbound_sockets=127.17.164.130
--tserver_master_addrs=127.17.164.190:44657
--webserver_port=33829
--webserver_interface=127.17.164.130
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 20:12:08.569137 24612 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:12:08.569626 24612 heartbeater.cc:461] Registering TS with master...
I20250622 20:12:08.570811 24612 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:12:08.575486 24425 ts_manager.cc:194] Registered new tserver with Master: 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129:41759)
I20250622 20:12:08.583300 24425 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.129:37763
I20250622 20:12:08.616751 24505 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Log is configured to *not* fsync() on all Append() calls
W20250622 20:12:08.838133 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:12:08.838605 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:12:08.839128 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:12:08.869443 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:12:08.870263 24616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.130
I20250622 20:12:08.904457 24616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.130:36941
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.164.130
--webserver_port=33829
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.130
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:12:08.905795 24616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:12:08.907357 24616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:12:08.923679 24624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:09.587659 24612 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
W20250622 20:12:08.924135 24623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:08.926437 24616 server_base.cc:1048] running on GCE node
W20250622 20:12:08.925047 24626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:10.075544 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:12:10.078142 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:12:10.079533 24616 hybrid_clock.cc:648] HybridClock initialized: now 1750623130079483 us; error 70 us; skew 500 ppm
I20250622 20:12:10.080283 24616 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:12:10.087040 24616 webserver.cc:469] Webserver started at http://127.17.164.130:33829/ using document root <none> and password file <none>
I20250622 20:12:10.087852 24616 fs_manager.cc:362] Metadata directory not provided
I20250622 20:12:10.088040 24616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:12:10.095667 24616 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250622 20:12:10.100350 24633 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:12:10.101375 24616 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250622 20:12:10.101647 24616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
uuid: "609af6f2666d47d09a79fc14319ce9ff"
format_stamp: "Formatted at 2025-06-22 20:11:53 on dist-test-slave-9mkf"
I20250622 20:12:10.103442 24616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:12:10.166944 24616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:12:10.168334 24616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:12:10.168730 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:12:10.171222 24616 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:12:10.177428 24640 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:12:10.187595 24616 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:12:10.187842 24616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.002s
I20250622 20:12:10.188102 24616 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:12:10.192472 24616 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:12:10.192651 24616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.000s
I20250622 20:12:10.193089 24640 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Bootstrap starting.
I20250622 20:12:10.359735 24616 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.130:36941
I20250622 20:12:10.359941 24746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.130:36941 every 8 connection(s)
I20250622 20:12:10.363327 24616 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb
I20250622 20:12:10.368544 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 24616
I20250622 20:12:10.370961 18066 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
/tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.164.131:33993
--local_ip_for_outbound_sockets=127.17.164.131
--tserver_master_addrs=127.17.164.190:44657
--webserver_port=34361
--webserver_interface=127.17.164.131
--builtin_ntp_servers=127.17.164.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250622 20:12:10.419898 24747 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:12:10.420385 24747 heartbeater.cc:461] Registering TS with master...
I20250622 20:12:10.421602 24747 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:12:10.425663 24425 ts_manager.cc:194] Registered new tserver with Master: 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941)
I20250622 20:12:10.428333 24425 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.130:60733
I20250622 20:12:10.538019 24640 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Log is configured to *not* fsync() on all Append() calls
I20250622 20:12:10.580463 24505 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:12:10.581583 24505 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Bootstrap complete.
I20250622 20:12:10.583298 24505 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent bootstrapping tablet: real 2.271s user 2.206s sys 0.056s
I20250622 20:12:10.598978 24505 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:10.601016 24505 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1635f5f2f402d849ed0bdbf527b2f, State: Initialized, Role: FOLLOWER
I20250622 20:12:10.601825 24505 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:10.606364 24505 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent starting tablet: real 0.023s user 0.021s sys 0.000s
W20250622 20:12:10.763645 24751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 20:12:10.764137 24751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 20:12:10.764662 24751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 20:12:10.796258 24751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 20:12:10.797113 24751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.164.131
I20250622 20:12:10.832497 24751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.164.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.164.131:33993
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.164.131
--webserver_port=34361
--tserver_master_addrs=127.17.164.190:44657
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.164.131
--log_dir=/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:57:46 UTC on 24a791456cd2
build id 6721
TSAN enabled
I20250622 20:12:10.833819 24751 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 20:12:10.835626 24751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 20:12:10.853154 24760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 20:12:11.432251 24747 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
I20250622 20:12:11.781455 24765 raft_consensus.cc:491] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:12:11.782119 24765 raft_consensus.cc:513] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:11.798027 24765 leader_election.cc:290] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941), 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
W20250622 20:12:11.805049 24499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.164.131:33993: connect: Connection refused (error 111)
W20250622 20:12:11.816005 24499 leader_election.cc:336] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): Network error: Client connection negotiation failed: client connection to 127.17.164.131:33993: connect: Connection refused (error 111)
I20250622 20:12:11.819732 24702 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff" is_pre_election: true
W20250622 20:12:11.838200 24500 leader_election.cc:343] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941): Illegal state: must be running to vote when last-logged opid is not known
I20250622 20:12:11.838856 24500 leader_election.cc:304] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f; no voters: 609af6f2666d47d09a79fc14319ce9ff, 7a56eca9c21c4b37822574bf53c547a3
I20250622 20:12:11.840473 24765 raft_consensus.cc:2747] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250622 20:12:10.856455 24762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:10.853816 24759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 20:12:12.274351 24761 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1418 milliseconds
I20250622 20:12:12.274479 24751 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250622 20:12:12.275692 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 20:12:12.278375 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 20:12:12.279793 24751 hybrid_clock.cc:648] HybridClock initialized: now 1750623132279745 us; error 67 us; skew 500 ppm
I20250622 20:12:12.280586 24751 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 20:12:12.287705 24751 webserver.cc:469] Webserver started at http://127.17.164.131:34361/ using document root <none> and password file <none>
I20250622 20:12:12.288601 24751 fs_manager.cc:362] Metadata directory not provided
I20250622 20:12:12.288859 24751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 20:12:12.296765 24751 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250622 20:12:12.301702 24774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 20:12:12.302738 24751 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250622 20:12:12.303038 24751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
uuid: "7a56eca9c21c4b37822574bf53c547a3"
format_stamp: "Formatted at 2025-06-22 20:11:55 on dist-test-slave-9mkf"
I20250622 20:12:12.304987 24751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 20:12:12.357026 24751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 20:12:12.358563 24751 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 20:12:12.359136 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 20:12:12.362293 24751 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 20:12:12.368891 24781 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 20:12:12.376464 24751 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 20:12:12.376690 24751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250622 20:12:12.377000 24751 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 20:12:12.383023 24751 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 20:12:12.383330 24751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.002s sys 0.003s
I20250622 20:12:12.385231 24781 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Bootstrap starting.
I20250622 20:12:12.615383 24751 rpc_server.cc:307] RPC server started. Bound to: 127.17.164.131:33993
I20250622 20:12:12.615589 24887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.164.131:33993 every 8 connection(s)
I20250622 20:12:12.619148 24751 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb
I20250622 20:12:12.625705 18066 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu as pid 24751
I20250622 20:12:12.670279 24888 heartbeater.cc:344] Connected to a master server at 127.17.164.190:44657
I20250622 20:12:12.670835 24888 heartbeater.cc:461] Registering TS with master...
I20250622 20:12:12.672236 24888 heartbeater.cc:507] Master 127.17.164.190:44657 requested a full tablet report, sending...
I20250622 20:12:12.677379 24424 ts_manager.cc:194] Registered new tserver with Master: 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:12:12.680395 24424 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.164.131:34755
I20250622 20:12:12.692828 18066 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250622 20:12:12.703405 24781 log.cc:826] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Log is configured to *not* fsync() on all Append() calls
I20250622 20:12:12.908625 24640 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:12:12.909451 24640 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Bootstrap complete.
I20250622 20:12:12.910679 24640 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Time spent bootstrapping tablet: real 2.718s user 2.601s sys 0.064s
I20250622 20:12:12.915478 24640 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:12.917335 24640 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609af6f2666d47d09a79fc14319ce9ff, State: Initialized, Role: FOLLOWER
I20250622 20:12:12.918031 24640 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:12.920982 24640 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20250622 20:12:13.422197 24901 raft_consensus.cc:491] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:12:13.422569 24901 raft_consensus.cc:513] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:13.423955 24901 leader_election.cc:290] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941), 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:12:13.424865 24702 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff" is_pre_election: true
I20250622 20:12:13.425618 24702 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 1.
I20250622 20:12:13.427138 24500 leader_election.cc:304] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f, 609af6f2666d47d09a79fc14319ce9ff; no voters:
I20250622 20:12:13.428161 24901 raft_consensus.cc:2802] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250622 20:12:13.428612 24901 raft_consensus.cc:491] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:12:13.429056 24901 raft_consensus.cc:3058] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:12:13.440943 24901 raft_consensus.cc:513] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:13.442566 24901 leader_election.cc:290] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 election: Requested vote from peers 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941), 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:12:13.443280 24702 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff"
I20250622 20:12:13.443691 24702 raft_consensus.cc:3058] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Advancing to term 2
I20250622 20:12:13.451424 24702 raft_consensus.cc:2466] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 2.
I20250622 20:12:13.452152 24500 leader_election.cc:304] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f, 609af6f2666d47d09a79fc14319ce9ff; no voters:
I20250622 20:12:13.452730 24901 raft_consensus.cc:2802] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 FOLLOWER]: Leader election won for term 2
I20250622 20:12:13.449862 24843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7a56eca9c21c4b37822574bf53c547a3" is_pre_election: true
I20250622 20:12:13.450019 24842 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8aa7a0acc08741378194be66d49845bd" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "7a56eca9c21c4b37822574bf53c547a3"
I20250622 20:12:13.460804 24901 raft_consensus.cc:695] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 LEADER]: Becoming Leader. State: Replica: 34d1635f5f2f402d849ed0bdbf527b2f, State: Running, Role: LEADER
W20250622 20:12:13.462011 24499 leader_election.cc:343] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): Illegal state: must be running to vote when last-logged opid is not known
I20250622 20:12:13.461772 24901 consensus_queue.cc:237] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
W20250622 20:12:13.462895 24499 leader_election.cc:343] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): Illegal state: must be running to vote when last-logged opid is not known
I20250622 20:12:13.471966 24424 catalog_manager.cc:5582] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f reported cstate change: term changed from 0 to 2, leader changed from <none> to 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129), VOTER 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129) added, VOTER 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130) added, VOTER 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131) added. New cstate: current_term: 2 leader_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } health_report { overall_health: UNKNOWN } } }
I20250622 20:12:13.685344 24888 heartbeater.cc:499] Master 127.17.164.190:44657 was elected leader, sending a full tablet report...
W20250622 20:12:13.862236 24499 consensus_peers.cc:487] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f -> Peer 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): Couldn't send request to peer 7a56eca9c21c4b37822574bf53c547a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250622 20:12:14.065217 24702 raft_consensus.cc:1273] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Refusing update from remote peer 34d1635f5f2f402d849ed0bdbf527b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250622 20:12:14.067546 24901 consensus_queue.cc:1035] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250622 20:12:14.071954 18066 scanner-internal.cc:458] Time spent opening tablet: real 1.343s user 0.005s sys 0.001s
I20250622 20:12:14.157842 24567 consensus_queue.cc:237] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } }
I20250622 20:12:14.163065 24702 raft_consensus.cc:1273] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Refusing update from remote peer 34d1635f5f2f402d849ed0bdbf527b2f: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250622 20:12:14.164613 24917 consensus_queue.cc:1035] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250622 20:12:14.170985 24917 raft_consensus.cc:2953] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } }
I20250622 20:12:14.173287 24702 raft_consensus.cc:2953] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } }
I20250622 20:12:14.189385 24411 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8aa7a0acc08741378194be66d49845bd with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 20:12:14.195001 24424 catalog_manager.cc:5582] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f reported cstate change: config changed from index -1 to 207, VOTER 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131) evicted. New cstate: current_term: 2 leader_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } health_report { overall_health: HEALTHY } } }
I20250622 20:12:14.233978 24567 consensus_queue.cc:237] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } }
I20250622 20:12:14.236855 24917 raft_consensus.cc:2953] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } }
I20250622 20:12:14.245350 24411 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8aa7a0acc08741378194be66d49845bd with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250622 20:12:14.251185 24424 catalog_manager.cc:5582] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f reported cstate change: config changed from index 207 to 208, VOTER 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130) evicted. New cstate: current_term: 2 leader_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } health_report { overall_health: HEALTHY } } }
I20250622 20:12:14.301515 24823 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa7a0acc08741378194be66d49845bd with delete_type TABLET_DATA_TOMBSTONED (TS 7a56eca9c21c4b37822574bf53c547a3 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:36724
I20250622 20:12:14.312258 24682 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa7a0acc08741378194be66d49845bd with delete_type TABLET_DATA_TOMBSTONED (TS 609af6f2666d47d09a79fc14319ce9ff not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:37722
W20250622 20:12:14.320225 24409 catalog_manager.cc:4908] TS 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): delete failed for tablet 8aa7a0acc08741378194be66d49845bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aa7a0acc08741378194be66d49845bd already in progress: opening tablet
I20250622 20:12:14.325392 24928 tablet_replica.cc:331] stopping tablet replica
I20250622 20:12:14.326308 24928 raft_consensus.cc:2241] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Raft consensus shutting down.
I20250622 20:12:14.327008 24928 raft_consensus.cc:2270] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 20:12:14.355002 24928 ts_tablet_manager.cc:1905] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 20:12:14.394269 24928 ts_tablet_manager.cc:1918] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250622 20:12:14.398257 24928 log.cc:1199] T 8aa7a0acc08741378194be66d49845bd P 609af6f2666d47d09a79fc14319ce9ff: Deleting WAL directory at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/wal/wals/8aa7a0acc08741378194be66d49845bd
I20250622 20:12:14.400209 24410 catalog_manager.cc:4928] TS 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941): tablet 8aa7a0acc08741378194be66d49845bd (table pre_rebuild [id=6857e9d3b33d4d11ac2a7c2310eeb8b6]) successfully deleted
I20250622 20:12:14.924058 24547 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:12:14.961110 24682 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:12:14.986862 24823 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:12:15.029582 24781 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250622 20:12:15.030644 24781 tablet_bootstrap.cc:492] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Bootstrap complete.
I20250622 20:12:15.032424 24781 ts_tablet_manager.cc:1397] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Time spent bootstrapping tablet: real 2.648s user 2.553s sys 0.044s
I20250622 20:12:15.040143 24781 raft_consensus.cc:357] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.043287 24781 raft_consensus.cc:738] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a56eca9c21c4b37822574bf53c547a3, State: Initialized, Role: FOLLOWER
I20250622 20:12:15.044342 24781 consensus_queue.cc:260] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.053714 24781 ts_tablet_manager.cc:1428] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20250622 20:12:15.060456 24823 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa7a0acc08741378194be66d49845bd with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:36724
I20250622 20:12:15.072271 24956 tablet_replica.cc:331] stopping tablet replica
I20250622 20:12:15.073115 24956 raft_consensus.cc:2241] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250622 20:12:15.073649 24956 raft_consensus.cc:2270] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250622 20:12:15.103711 24956 ts_tablet_manager.cc:1905] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250622 20:12:15.120697 24956 ts_tablet_manager.cc:1918] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250622 20:12:15.121130 24956 log.cc:1199] T 8aa7a0acc08741378194be66d49845bd P 7a56eca9c21c4b37822574bf53c547a3: Deleting WAL directory at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/wal/wals/8aa7a0acc08741378194be66d49845bd
I20250622 20:12:15.122845 24409 catalog_manager.cc:4928] TS 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993): tablet 8aa7a0acc08741378194be66d49845bd (table pre_rebuild [id=6857e9d3b33d4d11ac2a7c2310eeb8b6]) successfully deleted
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
088253faf14e40618a2efb47083b3156 | 127.17.164.190:44657 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:37669 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
34d1635f5f2f402d849ed0bdbf527b2f | 127.17.164.129:41759 | HEALTHY | <none> | 1 | 0
609af6f2666d47d09a79fc14319ce9ff | 127.17.164.130:36941 | HEALTHY | <none> | 0 | 0
7a56eca9c21c4b37822574bf53c547a3 | 127.17.164.131:33993 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.164.129 | experimental | 127.17.164.129:41759
local_ip_for_outbound_sockets | 127.17.164.130 | experimental | 127.17.164.130:36941
local_ip_for_outbound_sockets | 127.17.164.131 | experimental | 127.17.164.131:33993
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb | hidden | 127.17.164.129:41759
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb | hidden | 127.17.164.130:36941
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb | hidden | 127.17.164.131:33993
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:37669 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 20:12:15.259325 18066 log_verifier.cc:126] Checking tablet 8aa7a0acc08741378194be66d49845bd
I20250622 20:12:15.522471 18066 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 8aa7a0acc08741378194be66d49845bd
I20250622 20:12:15.524657 24425 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54880:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 20:12:15.525518 24425 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54880:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250622 20:12:15.550822 24425 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 088253faf14e40618a2efb47083b3156: Sending DeleteTablet for 1 replicas of tablet 8aa7a0acc08741378194be66d49845bd
I20250622 20:12:15.553110 18066 test_util.cc:276] Using random seed: -1239329221
I20250622 20:12:15.553033 24547 tablet_service.cc:1515] Processing DeleteTablet for tablet 8aa7a0acc08741378194be66d49845bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-22 20:12:15 UTC) from {username='slave'} at 127.0.0.1:56204
I20250622 20:12:15.557397 24962 tablet_replica.cc:331] stopping tablet replica
I20250622 20:12:15.558431 24962 raft_consensus.cc:2241] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 LEADER]: Raft consensus shutting down.
I20250622 20:12:15.559077 24962 raft_consensus.cc:2270] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250622 20:12:15.591053 24962 ts_tablet_manager.cc:1905] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250622 20:12:15.602736 24425 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46256:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 20:12:15.606251 24962 ts_tablet_manager.cc:1918] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
W20250622 20:12:15.606511 24425 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250622 20:12:15.606695 24962 log.cc:1199] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Deleting WAL directory at /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/wal/wals/8aa7a0acc08741378194be66d49845bd
I20250622 20:12:15.607537 24962 ts_tablet_manager.cc:1939] T 8aa7a0acc08741378194be66d49845bd P 34d1635f5f2f402d849ed0bdbf527b2f: Deleting consensus metadata
I20250622 20:12:15.610502 24411 catalog_manager.cc:4928] TS 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129:41759): tablet 8aa7a0acc08741378194be66d49845bd (table pre_rebuild [id=6857e9d3b33d4d11ac2a7c2310eeb8b6]) successfully deleted
I20250622 20:12:15.638078 24823 tablet_service.cc:1468] Processing CreateTablet for tablet 7a49409295c440718ecfb46d97cb2df8 (DEFAULT_TABLE table=post_rebuild [id=3269e1b09c814bc99cb6ce7e368fa757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:12:15.638988 24682 tablet_service.cc:1468] Processing CreateTablet for tablet 7a49409295c440718ecfb46d97cb2df8 (DEFAULT_TABLE table=post_rebuild [id=3269e1b09c814bc99cb6ce7e368fa757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:12:15.639386 24823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a49409295c440718ecfb46d97cb2df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:12:15.640144 24682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a49409295c440718ecfb46d97cb2df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:12:15.645144 24547 tablet_service.cc:1468] Processing CreateTablet for tablet 7a49409295c440718ecfb46d97cb2df8 (DEFAULT_TABLE table=post_rebuild [id=3269e1b09c814bc99cb6ce7e368fa757]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 20:12:15.646397 24547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a49409295c440718ecfb46d97cb2df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 20:12:15.677680 24969 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: Bootstrap starting.
I20250622 20:12:15.677867 24970 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: Bootstrap starting.
I20250622 20:12:15.684613 24969 tablet_bootstrap.cc:654] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: Neither blocks nor log segments found. Creating new log.
I20250622 20:12:15.681619 24971 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: Bootstrap starting.
I20250622 20:12:15.690822 24971 tablet_bootstrap.cc:654] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: Neither blocks nor log segments found. Creating new log.
I20250622 20:12:15.692860 24970 tablet_bootstrap.cc:654] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: Neither blocks nor log segments found. Creating new log.
I20250622 20:12:15.701316 24970 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: No bootstrap required, opened a new log
I20250622 20:12:15.701850 24970 ts_tablet_manager.cc:1397] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.000s
I20250622 20:12:15.704757 24970 raft_consensus.cc:357] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.706640 24971 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: No bootstrap required, opened a new log
I20250622 20:12:15.707049 24971 ts_tablet_manager.cc:1397] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.005s
I20250622 20:12:15.709583 24971 raft_consensus.cc:357] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.710196 24970 raft_consensus.cc:383] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:12:15.710315 24971 raft_consensus.cc:383] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:12:15.710496 24970 raft_consensus.cc:738] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609af6f2666d47d09a79fc14319ce9ff, State: Initialized, Role: FOLLOWER
I20250622 20:12:15.710716 24971 raft_consensus.cc:738] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1635f5f2f402d849ed0bdbf527b2f, State: Initialized, Role: FOLLOWER
I20250622 20:12:15.711436 24969 tablet_bootstrap.cc:492] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: No bootstrap required, opened a new log
I20250622 20:12:15.711110 24970 consensus_queue.cc:260] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.711980 24969 ts_tablet_manager.cc:1397] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.031s
I20250622 20:12:15.713259 24971 consensus_queue.cc:260] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.714819 24969 raft_consensus.cc:357] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.715834 24969 raft_consensus.cc:383] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 20:12:15.716243 24969 raft_consensus.cc:738] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a56eca9c21c4b37822574bf53c547a3, State: Initialized, Role: FOLLOWER
I20250622 20:12:15.717218 24969 consensus_queue.cc:260] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.721189 24970 ts_tablet_manager.cc:1428] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20250622 20:12:15.723374 24969 ts_tablet_manager.cc:1428] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250622 20:12:15.726601 24971 ts_tablet_manager.cc:1428] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: Time spent starting tablet: real 0.019s user 0.003s sys 0.007s
I20250622 20:12:15.760757 24976 raft_consensus.cc:491] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 20:12:15.761206 24976 raft_consensus.cc:513] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.763005 24976 leader_election.cc:290] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941), 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:12:15.763727 24702 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a49409295c440718ecfb46d97cb2df8" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff" is_pre_election: true
I20250622 20:12:15.764335 24702 raft_consensus.cc:2466] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 0.
I20250622 20:12:15.764535 24843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a49409295c440718ecfb46d97cb2df8" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a56eca9c21c4b37822574bf53c547a3" is_pre_election: true
I20250622 20:12:15.765251 24843 raft_consensus.cc:2466] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 0.
I20250622 20:12:15.765409 24500 leader_election.cc:304] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f, 609af6f2666d47d09a79fc14319ce9ff; no voters:
I20250622 20:12:15.766099 24976 raft_consensus.cc:2802] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 20:12:15.766420 24976 raft_consensus.cc:491] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 20:12:15.766664 24976 raft_consensus.cc:3058] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:12:15.770973 24976 raft_consensus.cc:513] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.772270 24976 leader_election.cc:290] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 1 election: Requested vote from peers 609af6f2666d47d09a79fc14319ce9ff (127.17.164.130:36941), 7a56eca9c21c4b37822574bf53c547a3 (127.17.164.131:33993)
I20250622 20:12:15.773106 24702 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a49409295c440718ecfb46d97cb2df8" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "609af6f2666d47d09a79fc14319ce9ff"
I20250622 20:12:15.773283 24843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a49409295c440718ecfb46d97cb2df8" candidate_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a56eca9c21c4b37822574bf53c547a3"
I20250622 20:12:15.773494 24702 raft_consensus.cc:3058] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:12:15.773782 24843 raft_consensus.cc:3058] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 0 FOLLOWER]: Advancing to term 1
I20250622 20:12:15.777509 24702 raft_consensus.cc:2466] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 1.
I20250622 20:12:15.778292 24500 leader_election.cc:304] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d1635f5f2f402d849ed0bdbf527b2f, 609af6f2666d47d09a79fc14319ce9ff; no voters:
I20250622 20:12:15.778908 24976 raft_consensus.cc:2802] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 FOLLOWER]: Leader election won for term 1
I20250622 20:12:15.780097 24976 raft_consensus.cc:695] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [term 1 LEADER]: Becoming Leader. State: Replica: 34d1635f5f2f402d849ed0bdbf527b2f, State: Running, Role: LEADER
I20250622 20:12:15.780239 24843 raft_consensus.cc:2466] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d1635f5f2f402d849ed0bdbf527b2f in term 1.
I20250622 20:12:15.780836 24976 consensus_queue.cc:237] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } }
I20250622 20:12:15.788596 24423 catalog_manager.cc:5582] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d1635f5f2f402d849ed0bdbf527b2f (127.17.164.129). New cstate: current_term: 1 leader_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34d1635f5f2f402d849ed0bdbf527b2f" member_type: VOTER last_known_addr { host: "127.17.164.129" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 } health_report { overall_health: UNKNOWN } } }
W20250622 20:12:15.821112 24613 tablet.cc:2378] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:12:15.881939 24889 tablet.cc:2378] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250622 20:12:15.910565 24748 tablet.cc:2378] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 20:12:15.948525 24702 raft_consensus.cc:1273] T 7a49409295c440718ecfb46d97cb2df8 P 609af6f2666d47d09a79fc14319ce9ff [term 1 FOLLOWER]: Refusing update from remote peer 34d1635f5f2f402d849ed0bdbf527b2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:12:15.948576 24843 raft_consensus.cc:1273] T 7a49409295c440718ecfb46d97cb2df8 P 7a56eca9c21c4b37822574bf53c547a3 [term 1 FOLLOWER]: Refusing update from remote peer 34d1635f5f2f402d849ed0bdbf527b2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 20:12:15.950279 24978 consensus_queue.cc:1035] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "609af6f2666d47d09a79fc14319ce9ff" member_type: VOTER last_known_addr { host: "127.17.164.130" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250622 20:12:15.951107 24976 consensus_queue.cc:1035] T 7a49409295c440718ecfb46d97cb2df8 P 34d1635f5f2f402d849ed0bdbf527b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a56eca9c21c4b37822574bf53c547a3" member_type: VOTER last_known_addr { host: "127.17.164.131" port: 33993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 20:12:15.984169 24986 mvcc.cc:204] Tried to move back new op lower bound from 7170552364830851072 to 7170552364164796416. Current Snapshot: MvccSnapshot[applied={T|T < 7170552364830851072}]
I20250622 20:12:15.994478 24987 mvcc.cc:204] Tried to move back new op lower bound from 7170552364830851072 to 7170552364164796416. Current Snapshot: MvccSnapshot[applied={T|T < 7170552364830851072}]
I20250622 20:12:20.669137 24682 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 20:12:20.677778 24547 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 20:12:20.677966 24823 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
088253faf14e40618a2efb47083b3156 | 127.17.164.190:44657 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:37669 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
34d1635f5f2f402d849ed0bdbf527b2f | 127.17.164.129:41759 | HEALTHY | <none> | 1 | 0
609af6f2666d47d09a79fc14319ce9ff | 127.17.164.130:36941 | HEALTHY | <none> | 0 | 0
7a56eca9c21c4b37822574bf53c547a3 | 127.17.164.131:33993 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.17.164.129 | experimental | 127.17.164.129:41759
local_ip_for_outbound_sockets | 127.17.164.130 | experimental | 127.17.164.130:36941
local_ip_for_outbound_sockets | 127.17.164.131 | experimental | 127.17.164.131:33993
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-0/data/info.pb | hidden | 127.17.164.129:41759
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-1/data/info.pb | hidden | 127.17.164.130:36941
server_dump_info_path | /tmp/dist-test-taskRvQqZa/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750622963291160-18066-0/minicluster-data/ts-2/data/info.pb | hidden | 127.17.164.131:33993
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.17.164.148:37669 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250622 20:12:20.913232 18066 log_verifier.cc:126] Checking tablet 7a49409295c440718ecfb46d97cb2df8
I20250622 20:12:21.653551 18066 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 7a49409295c440718ecfb46d97cb2df8
I20250622 20:12:21.654389 18066 log_verifier.cc:126] Checking tablet 8aa7a0acc08741378194be66d49845bd
I20250622 20:12:21.654683 18066 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 8aa7a0acc08741378194be66d49845bd
I20250622 20:12:21.664812 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 24462
I20250622 20:12:21.705684 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 24616
I20250622 20:12:21.741353 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 24751
I20250622 20:12:21.777241 18066 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRvQqZa/build/tsan/bin/kudu with pid 24392
2025-06-22T20:12:21Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (33667 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (33667 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (178471 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST