[==========] Running 6 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from RaftConfigChangeITest
[ RUN ] RaftConfigChangeITest.TestKudu2147
2026-11-20T15:00:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:00:45Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261120 15:00:45.777485 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:32829
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:32829 with env {}
W20261120 15:00:45.879487 6857 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:45.879644 6857 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:45.879721 6857 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:45.883100 6857 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:00:45.883154 6857 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:45.883175 6857 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:00:45.883194 6857 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:00:45.887066 6857 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:32829
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:45.887858 6857 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:45.888595 6857 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:45.894081 6863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:45.894143 6865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:45.894207 6862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:45.894289 6857 server_base.cc:1061] running on GCE node
I20261120 15:00:45.894766 6857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:45.895552 6857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:45.896733 6857 hybrid_clock.cc:648] HybridClock initialized: now 1795186845896667 us; error 80 us; skew 500 ppm
I20261120 15:00:45.898351 6857 webserver.cc:577] Webserver started at http://127.6.176.126:42371/ using document root <none> and password file <none>
I20261120 15:00:45.898912 6857 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:45.898983 6857 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:45.899190 6857 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:45.900508 6857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data/instance:
uuid: "c6b3fa09987c41d78870cd9160fa3d0d"
format_stamp: "Formatted at 2026-11-20 15:00:45 on dist-test-slave-m239"
I20261120 15:00:45.900964 6857 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal/instance:
uuid: "c6b3fa09987c41d78870cd9160fa3d0d"
format_stamp: "Formatted at 2026-11-20 15:00:45 on dist-test-slave-m239"
I20261120 15:00:45.904079 6857 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:00:45.906172 6871 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:45.907069 6857 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:00:45.907212 6857 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal
uuid: "c6b3fa09987c41d78870cd9160fa3d0d"
format_stamp: "Formatted at 2026-11-20 15:00:45 on dist-test-slave-m239"
I20261120 15:00:45.907308 6857 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:45.921564 6857 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:45.922309 6857 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:45.922497 6857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:45.930281 6857 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:32829
I20261120 15:00:45.930279 6923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:32829 every 8 connection(s)
I20261120 15:00:45.931433 6857 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
I20261120 15:00:45.932679 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 6857
I20261120 15:00:45.932832 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/master-0/wal/instance
I20261120 15:00:45.934670 6924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:45.940480 6924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d: Bootstrap starting.
I20261120 15:00:45.942668 6924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:45.943557 6924 log.cc:826] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:45.945046 6924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d: No bootstrap required, opened a new log
I20261120 15:00:45.947525 6924 raft_consensus.cc:366] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } }
I20261120 15:00:45.947721 6924 raft_consensus.cc:392] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:45.947801 6924 raft_consensus.cc:787] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b3fa09987c41d78870cd9160fa3d0d, State: Initialized, Role: FOLLOWER
I20261120 15:00:45.948238 6924 consensus_queue.cc:261] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } }
I20261120 15:00:45.948393 6924 raft_consensus.cc:407] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:00:45.948478 6924 raft_consensus.cc:534] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:00:45.948568 6924 raft_consensus.cc:3296] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:45.948875 6924 raft_consensus.cc:556] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } }
I20261120 15:00:45.949815 6924 leader_election.cc:304] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b3fa09987c41d78870cd9160fa3d0d; no voters:
I20261120 15:00:45.950052 6924 leader_election.cc:290] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:00:45.950100 6929 raft_consensus.cc:3024] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:00:45.950320 6929 raft_consensus.cc:744] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [term 1 LEADER]: Becoming Leader. State: Replica: c6b3fa09987c41d78870cd9160fa3d0d, State: Running, Role: LEADER
I20261120 15:00:45.950680 6929 consensus_queue.cc:238] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } }
I20261120 15:00:45.951159 6924 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:00:45.952566 6931 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } } }
I20261120 15:00:45.952714 6931 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:45.953044 6930 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b3fa09987c41d78870cd9160fa3d0d. Latest consensus state: current_term: 1 leader_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b3fa09987c41d78870cd9160fa3d0d" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 32829 } } }
I20261120 15:00:45.953148 6930 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:45.953626 6938 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:00:45.956883 6938 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:00:45.962204 6938 catalog_manager.cc:1398] Generated new cluster ID: 95eb9d36f59440f0b1d9a685fee030db
I20261120 15:00:45.962280 6938 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:00:45.980890 6938 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:00:45.981758 6938 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:00:45.990978 6938 catalog_manager.cc:6237] T 00000000000000000000000000000000 P c6b3fa09987c41d78870cd9160fa3d0d: Generated new TSK 0
I20261120 15:00:45.991703 6938 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:00:45.995420 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--builtin_ntp_servers=127.6.176.84:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
W20261120 15:00:46.098384 6948 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:46.098591 6948 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:46.098677 6948 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:46.102176 6948 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:46.102284 6948 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:00:46.106478 6948 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:46.107266 6948 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:46.108027 6948 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:46.113538 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:46.113608 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:46.113528 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:46.114159 6948 server_base.cc:1061] running on GCE node
I20261120 15:00:46.114617 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:46.115272 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:46.116444 6948 hybrid_clock.cc:648] HybridClock initialized: now 1795186846116426 us; error 39 us; skew 500 ppm
I20261120 15:00:46.118260 6948 webserver.cc:577] Webserver started at http://127.6.176.65:42777/ using document root <none> and password file <none>
I20261120 15:00:46.118893 6948 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:46.118957 6948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:46.119177 6948 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:46.120872 6948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data/instance:
uuid: "80de259c53e54534bd2bc6221202c31d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.121363 6948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance:
uuid: "80de259c53e54534bd2bc6221202c31d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.125267 6948 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20261120 15:00:46.127967 6962 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.129057 6948 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:00:46.129165 6948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal
uuid: "80de259c53e54534bd2bc6221202c31d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.129258 6948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:46.146090 6948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:46.146554 6948 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:46.146680 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:46.147125 6948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:46.148102 6948 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:46.148150 6948 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.148190 6948 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:46.148205 6948 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.157332 6948 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:34207
I20261120 15:00:46.157364 7075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:34207 every 8 connection(s)
I20261120 15:00:46.158339 6948 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
I20261120 15:00:46.161229 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 6948
I20261120 15:00:46.161345 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance
I20261120 15:00:46.163537 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--builtin_ntp_servers=127.6.176.84:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
I20261120 15:00:46.167254 7076 heartbeater.cc:345] Connected to a master server at 127.6.176.126:32829
I20261120 15:00:46.167552 7076 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:46.168265 7076 heartbeater.cc:508] Master 127.6.176.126:32829 requested a full tablet report, sending...
I20261120 15:00:46.170279 6888 ts_manager.cc:210] Registered new tserver with Master: 80de259c53e54534bd2bc6221202c31d (127.6.176.65:34207)
I20261120 15:00:46.172231 6888 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:36479
W20261120 15:00:46.265604 7079 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:46.265769 7079 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:46.265811 7079 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:46.269116 7079 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:46.269217 7079 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:00:46.273176 7079 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:46.273913 7079 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:46.274567 7079 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:46.280122 7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:46.280122 7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:46.280148 7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:46.280586 7079 server_base.cc:1061] running on GCE node
I20261120 15:00:46.280941 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:46.281505 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:46.282680 7079 hybrid_clock.cc:648] HybridClock initialized: now 1795186846282673 us; error 28 us; skew 500 ppm
I20261120 15:00:46.284310 7079 webserver.cc:577] Webserver started at http://127.6.176.66:32841/ using document root <none> and password file <none>
I20261120 15:00:46.284849 7079 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:46.284921 7079 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:46.285164 7079 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:46.286399 7079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data/instance:
uuid: "ef71d464c44248dd8c8cb31eeb46082d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.286835 7079 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance:
uuid: "ef71d464c44248dd8c8cb31eeb46082d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.289976 7079 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:00:46.292116 7093 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.293177 7079 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:00:46.293304 7079 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal
uuid: "ef71d464c44248dd8c8cb31eeb46082d"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.293401 7079 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:46.307848 7079 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:46.308355 7079 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:46.308528 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:46.309028 7079 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:46.310012 7079 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:46.310060 7079 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.310109 7079 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:46.310125 7079 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.319521 7079 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:36067
I20261120 15:00:46.319547 7206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:36067 every 8 connection(s)
I20261120 15:00:46.320554 7079 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
I20261120 15:00:46.327924 7207 heartbeater.cc:345] Connected to a master server at 127.6.176.126:32829
I20261120 15:00:46.328174 7207 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:46.328660 7207 heartbeater.cc:508] Master 127.6.176.126:32829 requested a full tablet report, sending...
I20261120 15:00:46.329646 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7079
I20261120 15:00:46.329713 6888 ts_manager.cc:210] Registered new tserver with Master: ef71d464c44248dd8c8cb31eeb46082d (127.6.176.66:36067)
I20261120 15:00:46.329747 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance
I20261120 15:00:46.330407 6888 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:33273
I20261120 15:00:46.332176 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--builtin_ntp_servers=127.6.176.84:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
W20261120 15:00:46.435539 7210 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:46.435706 7210 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:46.435751 7210 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:46.439168 7210 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:46.439273 7210 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:00:46.443185 7210 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:32829
--never_fsync=true
--heap_profile_path=/tmp/kudu.7210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:46.444015 7210 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:46.444694 7210 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:46.450610 7215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:46.450661 7210 server_base.cc:1061] running on GCE node
W20261120 15:00:46.450628 7216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:46.450603 7218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:46.451380 7210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:46.452013 7210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:46.453177 7210 hybrid_clock.cc:648] HybridClock initialized: now 1795186846453162 us; error 34 us; skew 500 ppm
I20261120 15:00:46.454846 7210 webserver.cc:577] Webserver started at http://127.6.176.67:36029/ using document root <none> and password file <none>
I20261120 15:00:46.455493 7210 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:46.455566 7210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:46.455794 7210 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:46.457089 7210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data/instance:
uuid: "242a650c70534482b9221def84d15f79"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.457542 7210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance:
uuid: "242a650c70534482b9221def84d15f79"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.460814 7210 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:00:46.463068 7224 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.464066 7210 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:00:46.464196 7210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal
uuid: "242a650c70534482b9221def84d15f79"
format_stamp: "Formatted at 2026-11-20 15:00:46 on dist-test-slave-m239"
I20261120 15:00:46.464295 7210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:46.481027 7210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:46.481542 7210 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:46.481719 7210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:46.482215 7210 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:46.483146 7210 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:46.483215 7210 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:46.483290 7210 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:46.483361 7210 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20261120 15:00:46.492952 7210 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:40147
I20261120 15:00:46.493062 7337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:40147 every 8 connection(s)
I20261120 15:00:46.493959 7210 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
I20261120 15:00:46.498606 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7210
I20261120 15:00:46.498701 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance
I20261120 15:00:46.501994 7338 heartbeater.cc:345] Connected to a master server at 127.6.176.126:32829
I20261120 15:00:46.502174 7338 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:46.502579 7338 heartbeater.cc:508] Master 127.6.176.126:32829 requested a full tablet report, sending...
I20261120 15:00:46.503605 6887 ts_manager.cc:210] Registered new tserver with Master: 242a650c70534482b9221def84d15f79 (127.6.176.67:40147)
I20261120 15:00:46.504127 6887 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:33055
I20261120 15:00:46.513616 6849 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20261120 15:00:46.534607 6887 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58582:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261120 15:00:46.535715 6887 catalog_manager.cc:7244] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261120 15:00:46.550787 7010 tablet_service.cc:1495] Processing CreateTablet for tablet deeaa0f210884ee0963fefb0a0aa8890 (DEFAULT_TABLE table=test-workload [id=2607b2f704ef46dba7efe4cef11156cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:46.550786 7141 tablet_service.cc:1495] Processing CreateTablet for tablet deeaa0f210884ee0963fefb0a0aa8890 (DEFAULT_TABLE table=test-workload [id=2607b2f704ef46dba7efe4cef11156cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:46.551712 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deeaa0f210884ee0963fefb0a0aa8890. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:46.551733 7141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deeaa0f210884ee0963fefb0a0aa8890. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:46.551666 7272 tablet_service.cc:1495] Processing CreateTablet for tablet deeaa0f210884ee0963fefb0a0aa8890 (DEFAULT_TABLE table=test-workload [id=2607b2f704ef46dba7efe4cef11156cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:46.552418 7272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deeaa0f210884ee0963fefb0a0aa8890. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:46.556872 7360 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Bootstrap starting.
I20261120 15:00:46.557876 7361 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Bootstrap starting.
I20261120 15:00:46.559641 7362 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Bootstrap starting.
I20261120 15:00:46.559708 7360 tablet_bootstrap.cc:654] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:46.559811 7361 tablet_bootstrap.cc:654] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:46.560501 7361 log.cc:826] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:46.560586 7360 log.cc:826] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:46.561865 7360 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: No bootstrap required, opened a new log
I20261120 15:00:46.561864 7362 tablet_bootstrap.cc:654] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:46.562014 7361 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: No bootstrap required, opened a new log
I20261120 15:00:46.562106 7360 ts_tablet_manager.cc:1449] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:00:46.562213 7361 ts_tablet_manager.cc:1449] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20261120 15:00:46.562716 7362 log.cc:826] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:46.564230 7362 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: No bootstrap required, opened a new log
I20261120 15:00:46.564417 7362 ts_tablet_manager.cc:1449] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:00:46.564905 7360 raft_consensus.cc:366] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.565244 7360 raft_consensus.cc:392] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:46.565325 7360 raft_consensus.cc:787] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80de259c53e54534bd2bc6221202c31d, State: Initialized, Role: FOLLOWER
I20261120 15:00:46.565342 7361 raft_consensus.cc:366] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.565582 7361 raft_consensus.cc:392] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:46.565641 7361 raft_consensus.cc:787] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef71d464c44248dd8c8cb31eeb46082d, State: Initialized, Role: FOLLOWER
I20261120 15:00:46.565770 7360 consensus_queue.cc:261] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.566031 7361 consensus_queue.cc:261] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.566463 7076 heartbeater.cc:500] Master 127.6.176.126:32829 was elected leader, sending a full tablet report...
I20261120 15:00:46.566656 7360 ts_tablet_manager.cc:1480] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20261120 15:00:46.567274 7361 ts_tablet_manager.cc:1480] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261120 15:00:46.567519 7362 raft_consensus.cc:366] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.567659 7207 heartbeater.cc:500] Master 127.6.176.126:32829 was elected leader, sending a full tablet report...
I20261120 15:00:46.567761 7362 raft_consensus.cc:392] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:46.567842 7362 raft_consensus.cc:787] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242a650c70534482b9221def84d15f79, State: Initialized, Role: FOLLOWER
I20261120 15:00:46.568204 7362 consensus_queue.cc:261] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:46.568811 7338 heartbeater.cc:500] Master 127.6.176.126:32829 was elected leader, sending a full tablet report...
I20261120 15:00:46.568962 7362 ts_tablet_manager.cc:1480] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20261120 15:00:46.571274 7208 tablet.cc:2554] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:00:46.659323 7077 tablet.cc:2554] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:00:46.744786 7339 tablet.cc:2554] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 15:00:47.671940 7369 raft_consensus.cc:534] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:00:47.672164 7369 raft_consensus.cc:556] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:47.673336 7369 leader_election.cc:290] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80de259c53e54534bd2bc6221202c31d (127.6.176.65:34207), ef71d464c44248dd8c8cb31eeb46082d (127.6.176.66:36067)
I20261120 15:00:47.677205 7030 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890" candidate_uuid: "242a650c70534482b9221def84d15f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80de259c53e54534bd2bc6221202c31d" is_pre_election: true
I20261120 15:00:47.677295 7161 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890" candidate_uuid: "242a650c70534482b9221def84d15f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef71d464c44248dd8c8cb31eeb46082d" is_pre_election: true
I20261120 15:00:47.677524 7030 raft_consensus.cc:2062] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 242a650c70534482b9221def84d15f79 in term 0
I20261120 15:00:47.677630 7161 raft_consensus.cc:2062] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 242a650c70534482b9221def84d15f79 in term 0
I20261120 15:00:47.678056 7228 leader_election.cc:304] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 242a650c70534482b9221def84d15f79, 80de259c53e54534bd2bc6221202c31d, ef71d464c44248dd8c8cb31eeb46082d; no voters:
I20261120 15:00:47.678318 7369 raft_consensus.cc:3024] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 15:00:47.678408 7369 raft_consensus.cc:534] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:00:47.678495 7369 raft_consensus.cc:3296] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:47.678797 7369 raft_consensus.cc:556] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:47.679986 7369 leader_election.cc:290] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [CANDIDATE]: Term 1 election: Requested vote from peers 80de259c53e54534bd2bc6221202c31d (127.6.176.65:34207), ef71d464c44248dd8c8cb31eeb46082d (127.6.176.66:36067)
I20261120 15:00:47.680317 7161 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890" candidate_uuid: "242a650c70534482b9221def84d15f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef71d464c44248dd8c8cb31eeb46082d"
I20261120 15:00:47.680418 7161 raft_consensus.cc:3296] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:47.680343 7030 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890" candidate_uuid: "242a650c70534482b9221def84d15f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80de259c53e54534bd2bc6221202c31d"
I20261120 15:00:47.680526 7030 raft_consensus.cc:3296] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:47.680831 7161 raft_consensus.cc:2062] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 242a650c70534482b9221def84d15f79 in term 1
I20261120 15:00:47.680902 7030 raft_consensus.cc:2062] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 242a650c70534482b9221def84d15f79 in term 1
I20261120 15:00:47.682034 7228 leader_election.cc:304] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242a650c70534482b9221def84d15f79, 80de259c53e54534bd2bc6221202c31d; no voters:
I20261120 15:00:47.682191 7369 raft_consensus.cc:3024] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:00:47.682277 7369 raft_consensus.cc:744] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 LEADER]: Becoming Leader. State: Replica: 242a650c70534482b9221def84d15f79, State: Running, Role: LEADER
I20261120 15:00:47.682621 7369 consensus_queue.cc:238] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:47.685776 7378 consensus_queue.cc:1049] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:00:47.686169 7378 consensus_queue.cc:1049] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:00:47.686051 6887 catalog_manager.cc:5856] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 242a650c70534482b9221def84d15f79 (127.6.176.67). New cstate: current_term: 1 leader_uuid: "242a650c70534482b9221def84d15f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } health_report { overall_health: HEALTHY } } }
W20261120 15:00:48.045914 7070 generic_service.cc:139] {username='slave'} at 127.0.0.1:41716 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261120 15:00:48.046017 7070 generic_service.cc:175] {username='slave'} at 127.0.0.1:41716 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261120 15:00:48.046775 7070 generic_service.cc:175] {username='slave'} at 127.0.0.1:41716 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
W20261120 15:00:48.047658 7201 generic_service.cc:139] {username='slave'} at 127.0.0.1:43492 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261120 15:00:48.047746 7201 generic_service.cc:175] {username='slave'} at 127.0.0.1:43492 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261120 15:00:48.048432 7201 generic_service.cc:175] {username='slave'} at 127.0.0.1:43492 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
W20261120 15:00:48.049124 7332 generic_service.cc:139] {username='slave'} at 127.0.0.1:50606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261120 15:00:48.049199 7332 generic_service.cc:175] {username='slave'} at 127.0.0.1:50606 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261120 15:00:48.049837 7332 generic_service.cc:175] {username='slave'} at 127.0.0.1:50606 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '2000'
I20261120 15:00:48.051051 7292 consensus_queue.cc:238] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:48.052150 7161 raft_consensus.cc:1338] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 1 FOLLOWER]: Refusing update from remote peer 242a650c70534482b9221def84d15f79: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20261120 15:00:48.052454 7369 consensus_queue.cc:1049] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20261120 15:00:48.053954 7378 raft_consensus.cc:3183] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } }
I20261120 15:00:48.054306 7161 raft_consensus.cc:3183] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } }
W20261120 15:00:48.056248 7332 generic_service.cc:139] {username='slave'} at 127.0.0.1:50606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261120 15:00:48.056322 7332 generic_service.cc:175] {username='slave'} at 127.0.0.1:50606 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'true'
I20261120 15:00:48.057025 7161 tablet_service.cc:2028] Received Run Leader Election RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890"
dest_uuid: "ef71d464c44248dd8c8cb31eeb46082d"
from {username='slave'} at 127.0.0.1:43492
I20261120 15:00:48.057163 7161 raft_consensus.cc:534] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20261120 15:00:48.057228 7161 raft_consensus.cc:3296] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:00:48.057343 7161 raft_consensus.cc:556] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:48.058563 7161 leader_election.cc:290] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [CANDIDATE]: Term 2 election: Requested vote from peers 242a650c70534482b9221def84d15f79 (127.6.176.67:40147)
I20261120 15:00:48.062072 7292 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "deeaa0f210884ee0963fefb0a0aa8890" candidate_uuid: "ef71d464c44248dd8c8cb31eeb46082d" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "242a650c70534482b9221def84d15f79"
I20261120 15:00:48.062263 7292 raft_consensus.cc:3291] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 LEADER]: Stepping down as leader of term 1
I20261120 15:00:48.062330 7292 raft_consensus.cc:787] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 242a650c70534482b9221def84d15f79, State: Running, Role: LEADER
I20261120 15:00:48.062512 7292 consensus_queue.cc:261] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
I20261120 15:00:48.062669 7292 raft_consensus.cc:3296] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:00:48.062862 7292 raft_consensus.cc:2062] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 2 FOLLOWER]: Leader election vote request: granting yes vote for candidate ef71d464c44248dd8c8cb31eeb46082d in term 2
I20261120 15:00:48.064251 7095 leader_election.cc:304] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 242a650c70534482b9221def84d15f79, ef71d464c44248dd8c8cb31eeb46082d; no voters:
I20261120 15:00:48.064502 7375 raft_consensus.cc:3024] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 2 FOLLOWER]: Leader election won for term 2
I20261120 15:00:48.064616 7375 raft_consensus.cc:744] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 2 LEADER]: Becoming Leader. State: Replica: ef71d464c44248dd8c8cb31eeb46082d, State: Running, Role: LEADER
I20261120 15:00:48.064905 7375 consensus_queue.cc:238] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } }
W20261120 15:00:48.066954 7095 consensus_peers.cc:597] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d -> Peer 242a650c70534482b9221def84d15f79 (127.6.176.67:40147): Couldn't send request to peer 242a650c70534482b9221def84d15f79. Error code: UNKNOWN_ERROR (1). Status: Illegal state: Rejected: --follower_reject_update_consensus_requests is set to true.. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:00:50.056007 6885 catalog_manager.cc:5856] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 reported cstate change: config changed from index -1 to 28, term changed from 1 to 2, leader changed from 242a650c70534482b9221def84d15f79 (127.6.176.67) to <none>, VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) evicted. New cstate: current_term: 2 committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } }
I20261120 15:00:50.059259 7010 tablet_service.cc:1542] Processing DeleteTablet for tablet deeaa0f210884ee0963fefb0a0aa8890 with delete_type TABLET_DATA_TOMBSTONED (TS 80de259c53e54534bd2bc6221202c31d not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:41702
I20261120 15:00:50.059793 7415 tablet_replica.cc:333] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: stopping tablet replica
I20261120 15:00:50.060093 7415 raft_consensus.cc:2440] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20261120 15:00:50.060338 7415 raft_consensus.cc:2469] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20261120 15:00:50.061340 7415 ts_tablet_manager.cc:1962] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261120 15:00:50.063201 7415 ts_tablet_manager.cc:1975] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20261120 15:00:50.063271 7415 log.cc:1199] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Deleting WAL directory at /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1795186845751551-6849-0/minicluster-data/ts-0/wal/wals/deeaa0f210884ee0963fefb0a0aa8890
I20261120 15:00:50.063983 6875 catalog_manager.cc:5081] TS 80de259c53e54534bd2bc6221202c31d (127.6.176.65:34207): tablet deeaa0f210884ee0963fefb0a0aa8890 (table test-workload [id=2607b2f704ef46dba7efe4cef11156cf]) successfully deleted
W20261120 15:00:50.099674 7332 generic_service.cc:139] {username='slave'} at 127.0.0.1:50606 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261120 15:00:50.099792 7332 generic_service.cc:175] {username='slave'} at 127.0.0.1:50606 changed flags via RPC: follower_reject_update_consensus_requests from 'true' to 'false'
I20261120 15:00:50.102496 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 1.28 <uninitialized op>
I20261120 15:00:50.205390 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 1.28 <uninitialized op>
I20261120 15:00:50.408123 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 1.28 <uninitialized op>
I20261120 15:00:50.710902 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 1.28 <uninitialized op>
I20261120 15:00:51.113647 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 2.29 <uninitialized op>
I20261120 15:00:51.616412 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 2.29 <uninitialized op>
I20261120 15:00:52.060762 6885 catalog_manager.cc:5856] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 reported cstate change: leader changed from <none> to ef71d464c44248dd8c8cb31eeb46082d (127.6.176.66). New cstate: current_term: 2 leader_uuid: "ef71d464c44248dd8c8cb31eeb46082d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } }
I20261120 15:00:52.219462 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 2.29 <uninitialized op>
I20261120 15:00:52.922415 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.29 2.29 <uninitialized op>
I20261120 15:00:53.059048 7161 consensus_queue.cc:238] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } }
I20261120 15:00:53.060803 7292 raft_consensus.cc:1338] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 2 FOLLOWER]: Refusing update from remote peer ef71d464c44248dd8c8cb31eeb46082d: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20261120 15:00:53.061329 7428 consensus_queue.cc:1049] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [LEADER]: Connected to new peer: Peer: permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20261120 15:00:53.063386 7429 raft_consensus.cc:3183] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } } }
I20261120 15:00:53.063867 7292 raft_consensus.cc:3183] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } } }
I20261120 15:00:53.064656 6873 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet deeaa0f210884ee0963fefb0a0aa8890 with cas_config_opid_index 28: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20261120 15:00:53.064769 7097 consensus_peers.cc:597] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d -> Peer 80de259c53e54534bd2bc6221202c31d (127.6.176.65:34207): Couldn't send request to peer 80de259c53e54534bd2bc6221202c31d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:00:53.725292 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.30 2.30 <uninitialized op>
I20261120 15:00:54.065503 6885 catalog_manager.cc:5856] T deeaa0f210884ee0963fefb0a0aa8890 P 242a650c70534482b9221def84d15f79 reported cstate change: config changed from index 28 to 30, NON_VOTER 80de259c53e54534bd2bc6221202c31d (127.6.176.65) added. New cstate: current_term: 2 leader_uuid: "ef71d464c44248dd8c8cb31eeb46082d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } } }
I20261120 15:00:54.628134 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.30 2.30 <uninitialized op>
I20261120 15:00:55.630963 6849 cluster_itest_util.cc:258] Not converged past 1 yet: 2.30 2.30 <uninitialized op>
I20261120 15:00:56.117756 7441 ts_tablet_manager.cc:979] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Initiating tablet copy from peer ef71d464c44248dd8c8cb31eeb46082d (127.6.176.66:36067)
I20261120 15:00:56.118342 7441 tablet_copy_client.cc:323] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.176.66:36067
I20261120 15:00:56.121827 7181 tablet_copy_service.cc:140] P ef71d464c44248dd8c8cb31eeb46082d: Received BeginTabletCopySession request for tablet deeaa0f210884ee0963fefb0a0aa8890 from peer 80de259c53e54534bd2bc6221202c31d ({username='slave'} at 127.6.176.65:34563)
I20261120 15:00:56.121971 7181 tablet_copy_service.cc:161] P ef71d464c44248dd8c8cb31eeb46082d: Beginning new tablet copy session on tablet deeaa0f210884ee0963fefb0a0aa8890 from peer 80de259c53e54534bd2bc6221202c31d at {username='slave'} at 127.6.176.65:34563: session id = 80de259c53e54534bd2bc6221202c31d-deeaa0f210884ee0963fefb0a0aa8890
I20261120 15:00:56.123211 7181 tablet_copy_source_session.cc:215] T deeaa0f210884ee0963fefb0a0aa8890 P ef71d464c44248dd8c8cb31eeb46082d: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 15:00:56.124642 7441 ts_tablet_manager.cc:1962] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261120 15:00:56.126561 7441 ts_tablet_manager.cc:1975] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.27
I20261120 15:00:56.126765 7441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deeaa0f210884ee0963fefb0a0aa8890. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:56.128162 7441 tablet_copy_client.cc:806] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet copy: Starting download of 0 data blocks...
I20261120 15:00:56.128332 7441 tablet_copy_client.cc:670] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet copy: Starting download of 1 WAL segments...
I20261120 15:00:56.130280 7441 tablet_copy_client.cc:538] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 15:00:56.131801 7441 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Bootstrap starting.
I20261120 15:00:56.172268 7441 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=1300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 15:00:56.172605 7441 tablet_bootstrap.cc:492] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Bootstrap complete.
I20261120 15:00:56.172802 7441 ts_tablet_manager.cc:1449] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Time spent bootstrapping tablet: real 0.041s user 0.040s sys 0.000s
I20261120 15:00:56.173307 7441 raft_consensus.cc:366] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } }
I20261120 15:00:56.173437 7441 raft_consensus.cc:787] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 80de259c53e54534bd2bc6221202c31d, State: Initialized, Role: LEARNER
I20261120 15:00:56.173604 7441 consensus_queue.cc:261] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "ef71d464c44248dd8c8cb31eeb46082d" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 36067 } } peers { permanent_uuid: "242a650c70534482b9221def84d15f79" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 40147 } } peers { permanent_uuid: "80de259c53e54534bd2bc6221202c31d" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 34207 } attrs { promote: true } }
I20261120 15:00:56.173975 7441 ts_tablet_manager.cc:1480] T deeaa0f210884ee0963fefb0a0aa8890 P 80de259c53e54534bd2bc6221202c31d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20261120 15:00:56.174677 7181 tablet_copy_service.cc:342] P ef71d464c44248dd8c8cb31eeb46082d: Request end of tablet copy session 80de259c53e54534bd2bc6221202c31d-deeaa0f210884ee0963fefb0a0aa8890 received from {username='slave'} at 127.6.176.65:34563
I20261120 15:00:56.174770 7181 tablet_copy_service.cc:434] P ef71d464c44248dd8c8cb31eeb46082d: ending tablet copy session 80de259c53e54534bd2bc6221202c31d-deeaa0f210884ee0963fefb0a0aa8890 on tablet deeaa0f210884ee0963fefb0a0aa8890 with peer 80de259c53e54534bd2bc6221202c31d
I20261120 15:00:56.634903 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 6948
I20261120 15:00:56.642336 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7079
I20261120 15:00:56.650416 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7210
I20261120 15:00:56.656665 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 6857
2026-11-20T15:00:56Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestKudu2147 (10917 ms)
[ RUN ] RaftConfigChangeITest.TestNonVoterPromotion
2026-11-20T15:00:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:00:56Z Disabled control of system clock
I20261120 15:00:56.686626 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:34189
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:34189
--raft_prepare_replacement_before_eviction=true
--catalog_manager_evict_excess_replicas=false with env {}
W20261120 15:00:56.786777 7450 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:56.786936 7450 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:56.786986 7450 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:56.790449 7450 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:00:56.790524 7450 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:56.790546 7450 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:00:56.790571 7450 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:00:56.794380 7450 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal
--catalog_manager_evict_excess_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:34189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:56.795077 7450 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:56.795749 7450 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:56.801076 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:56.801124 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:56.801116 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:56.801286 7450 server_base.cc:1061] running on GCE node
I20261120 15:00:56.801671 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:56.802382 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:56.803517 7450 hybrid_clock.cc:648] HybridClock initialized: now 1795186856803492 us; error 43 us; skew 500 ppm
I20261120 15:00:56.805145 7450 webserver.cc:577] Webserver started at http://127.6.176.126:33825/ using document root <none> and password file <none>
I20261120 15:00:56.805723 7450 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:56.805769 7450 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:56.805987 7450 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:56.807227 7450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data/instance:
uuid: "bb37957c7efe4e8f93071a89624a7d9a"
format_stamp: "Formatted at 2026-11-20 15:00:56 on dist-test-slave-m239"
I20261120 15:00:56.807734 7450 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal/instance:
uuid: "bb37957c7efe4e8f93071a89624a7d9a"
format_stamp: "Formatted at 2026-11-20 15:00:56 on dist-test-slave-m239"
I20261120 15:00:56.810949 7450 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20261120 15:00:56.813261 7464 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:56.814230 7450 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:00:56.814375 7450 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal
uuid: "bb37957c7efe4e8f93071a89624a7d9a"
format_stamp: "Formatted at 2026-11-20 15:00:56 on dist-test-slave-m239"
I20261120 15:00:56.814463 7450 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:56.836400 7450 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:56.836907 7450 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:56.837067 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:56.843895 7450 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:34189
I20261120 15:00:56.843916 7516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:34189 every 8 connection(s)
I20261120 15:00:56.844914 7450 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
I20261120 15:00:56.848038 7517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:56.851557 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7450
I20261120 15:00:56.851646 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/master-0/wal/instance
I20261120 15:00:56.853346 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a: Bootstrap starting.
I20261120 15:00:56.855768 7517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:56.856698 7517 log.cc:826] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:56.858184 7517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a: No bootstrap required, opened a new log
I20261120 15:00:56.860638 7517 raft_consensus.cc:366] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } }
I20261120 15:00:56.860800 7517 raft_consensus.cc:392] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:56.860885 7517 raft_consensus.cc:787] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb37957c7efe4e8f93071a89624a7d9a, State: Initialized, Role: FOLLOWER
I20261120 15:00:56.861282 7517 consensus_queue.cc:261] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } }
I20261120 15:00:56.861404 7517 raft_consensus.cc:407] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:00:56.861483 7517 raft_consensus.cc:534] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:00:56.861598 7517 raft_consensus.cc:3296] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:56.861894 7517 raft_consensus.cc:556] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } }
I20261120 15:00:56.862840 7517 leader_election.cc:304] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb37957c7efe4e8f93071a89624a7d9a; no voters:
I20261120 15:00:56.863077 7517 leader_election.cc:290] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:00:56.863144 7522 raft_consensus.cc:3024] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:00:56.863411 7522 raft_consensus.cc:744] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [term 1 LEADER]: Becoming Leader. State: Replica: bb37957c7efe4e8f93071a89624a7d9a, State: Running, Role: LEADER
I20261120 15:00:56.863860 7522 consensus_queue.cc:238] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } }
I20261120 15:00:56.864351 7517 sys_catalog.cc:565] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:00:56.865216 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb37957c7efe4e8f93071a89624a7d9a. Latest consensus state: current_term: 1 leader_uuid: "bb37957c7efe4e8f93071a89624a7d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } } }
I20261120 15:00:56.865332 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:56.866153 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb37957c7efe4e8f93071a89624a7d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb37957c7efe4e8f93071a89624a7d9a" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 34189 } } }
I20261120 15:00:56.866302 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:56.866937 7528 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:00:56.870285 7528 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:00:56.875679 7528 catalog_manager.cc:1398] Generated new cluster ID: 8e684c2b80544e7e9969121cd59f422c
I20261120 15:00:56.875741 7528 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:00:56.907224 7528 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:00:56.908111 7528 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:00:56.920392 7528 catalog_manager.cc:6237] T 00000000000000000000000000000000 P bb37957c7efe4e8f93071a89624a7d9a: Generated new TSK 0
I20261120 15:00:56.920998 7528 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:00:56.932772 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--builtin_ntp_servers=127.6.176.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:00:57.040591 7541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:57.040762 7541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:57.040840 7541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:57.044276 7541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:57.044414 7541 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:00:57.048276 7541 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--never_fsync=true
--heap_profile_path=/tmp/kudu.7541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:57.049041 7541 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:57.049757 7541 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:57.055241 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.055260 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.055251 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.055521 7541 server_base.cc:1061] running on GCE node
I20261120 15:00:57.055913 7541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:57.056440 7541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:57.057606 7541 hybrid_clock.cc:648] HybridClock initialized: now 1795186857057597 us; error 26 us; skew 500 ppm
I20261120 15:00:57.059197 7541 webserver.cc:577] Webserver started at http://127.6.176.65:37935/ using document root <none> and password file <none>
I20261120 15:00:57.059834 7541 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:57.059911 7541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:57.060138 7541 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:57.061395 7541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data/instance:
uuid: "a689928847f64cad850b8da6f8f2dafd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.061839 7541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance:
uuid: "a689928847f64cad850b8da6f8f2dafd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.064961 7541 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:00:57.067154 7555 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.068137 7541 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:00:57.068264 7541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal
uuid: "a689928847f64cad850b8da6f8f2dafd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.068359 7541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:57.098472 7541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:57.099028 7541 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:57.099231 7541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:57.099808 7541 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:57.100770 7541 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:57.100840 7541 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.100917 7541 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:57.100970 7541 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.109833 7541 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:34701
I20261120 15:00:57.109865 7668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:34701 every 8 connection(s)
I20261120 15:00:57.110782 7541 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
I20261120 15:00:57.117934 7669 heartbeater.cc:345] Connected to a master server at 127.6.176.126:34189
I20261120 15:00:57.118171 7669 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:57.118639 7669 heartbeater.cc:508] Master 127.6.176.126:34189 requested a full tablet report, sending...
I20261120 15:00:57.119707 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7541
I20261120 15:00:57.119822 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance
I20261120 15:00:57.120385 7481 ts_manager.cc:210] Registered new tserver with Master: a689928847f64cad850b8da6f8f2dafd (127.6.176.65:34701)
I20261120 15:00:57.121773 7481 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:54187
I20261120 15:00:57.122184 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--builtin_ntp_servers=127.6.176.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:00:57.222407 7672 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:57.222581 7672 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:57.222622 7672 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:57.226131 7672 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:57.226244 7672 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:00:57.230090 7672 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--never_fsync=true
--heap_profile_path=/tmp/kudu.7672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:57.230768 7672 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:57.231477 7672 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:57.236792 7680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.236896 7678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.236915 7677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.237330 7672 server_base.cc:1061] running on GCE node
I20261120 15:00:57.237622 7672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:57.238152 7672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:57.239302 7672 hybrid_clock.cc:648] HybridClock initialized: now 1795186857239291 us; error 23 us; skew 500 ppm
I20261120 15:00:57.240892 7672 webserver.cc:577] Webserver started at http://127.6.176.66:43099/ using document root <none> and password file <none>
I20261120 15:00:57.241460 7672 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:57.241506 7672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:57.241730 7672 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:57.243077 7672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data/instance:
uuid: "7346930d1d97432a9707c73df3aff288"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.243551 7672 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance:
uuid: "7346930d1d97432a9707c73df3aff288"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.246683 7672 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20261120 15:00:57.248904 7686 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.249886 7672 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:00:57.250041 7672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal
uuid: "7346930d1d97432a9707c73df3aff288"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.250149 7672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:57.274101 7672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:57.274673 7672 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:57.274865 7672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:57.275511 7672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:57.276527 7672 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:57.276599 7672 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.276676 7672 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:57.276721 7672 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.285899 7672 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:46329
I20261120 15:00:57.285924 7799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:46329 every 8 connection(s)
I20261120 15:00:57.286921 7672 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
I20261120 15:00:57.287528 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7672
I20261120 15:00:57.287640 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance
I20261120 15:00:57.290143 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--builtin_ntp_servers=127.6.176.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20261120 15:00:57.296543 7800 heartbeater.cc:345] Connected to a master server at 127.6.176.126:34189
I20261120 15:00:57.296818 7800 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:57.297457 7800 heartbeater.cc:508] Master 127.6.176.126:34189 requested a full tablet report, sending...
I20261120 15:00:57.298745 7481 ts_manager.cc:210] Registered new tserver with Master: 7346930d1d97432a9707c73df3aff288 (127.6.176.66:46329)
I20261120 15:00:57.299606 7481 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:52399
W20261120 15:00:57.398473 7803 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:57.398640 7803 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:57.398684 7803 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:57.402168 7803 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:57.402276 7803 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:00:57.406198 7803 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--never_fsync=true
--heap_profile_path=/tmp/kudu.7803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:57.406957 7803 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:57.407696 7803 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:57.413134 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.413131 7811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.413321 7803 server_base.cc:1061] running on GCE node
W20261120 15:00:57.413403 7809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.413769 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:57.414300 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:57.415416 7803 hybrid_clock.cc:648] HybridClock initialized: now 1795186857415394 us; error 46 us; skew 500 ppm
I20261120 15:00:57.417117 7803 webserver.cc:577] Webserver started at http://127.6.176.67:33267/ using document root <none> and password file <none>
I20261120 15:00:57.417647 7803 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:57.417693 7803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:57.417881 7803 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:57.419171 7803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data/instance:
uuid: "571f2655dabd4acb88f01d4f51af21dd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.419627 7803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance:
uuid: "571f2655dabd4acb88f01d4f51af21dd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.422818 7803 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20261120 15:00:57.425148 7817 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.426194 7803 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:00:57.426303 7803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal
uuid: "571f2655dabd4acb88f01d4f51af21dd"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.426419 7803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:57.444197 7803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:57.444690 7803 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:57.444860 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:57.445360 7803 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:57.446286 7803 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:57.446331 7803 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.446400 7803 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:57.446439 7803 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.455628 7803 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:41071
I20261120 15:00:57.455655 7930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:41071 every 8 connection(s)
I20261120 15:00:57.456624 7803 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
I20261120 15:00:57.464291 7931 heartbeater.cc:345] Connected to a master server at 127.6.176.126:34189
I20261120 15:00:57.464509 7931 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:57.464964 7931 heartbeater.cc:508] Master 127.6.176.126:34189 requested a full tablet report, sending...
I20261120 15:00:57.466358 7481 ts_manager.cc:210] Registered new tserver with Master: 571f2655dabd4acb88f01d4f51af21dd (127.6.176.67:41071)
I20261120 15:00:57.466444 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7803
I20261120 15:00:57.466576 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance
I20261120 15:00:57.467027 7481 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:53899
I20261120 15:00:57.468789 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:0
--local_ip_for_outbound_sockets=127.6.176.68
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--builtin_ntp_servers=127.6.176.84:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:00:57.569463 7934 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:57.569628 7934 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:57.569670 7934 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:57.573020 7934 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:57.573123 7934 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:00:57.577208 7934 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:34189
--never_fsync=true
--heap_profile_path=/tmp/kudu.7934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:57.577947 7934 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:57.578649 7934 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:57.584971 7940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:57.584980 7939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.585083 7934 server_base.cc:1061] running on GCE node
W20261120 15:00:57.584985 7942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:57.585544 7934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:57.586205 7934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:57.587419 7934 hybrid_clock.cc:648] HybridClock initialized: now 1795186857587399 us; error 33 us; skew 500 ppm
I20261120 15:00:57.589452 7934 webserver.cc:577] Webserver started at http://127.6.176.68:39781/ using document root <none> and password file <none>
I20261120 15:00:57.590131 7934 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:57.590219 7934 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:57.590464 7934 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:57.591953 7934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data/instance:
uuid: "8d501f0f24c04b78a19107f50163c497"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.592545 7934 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal/instance:
uuid: "8d501f0f24c04b78a19107f50163c497"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.595927 7934 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20261120 15:00:57.598279 7948 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.599323 7934 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:00:57.599541 7934 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal
uuid: "8d501f0f24c04b78a19107f50163c497"
format_stamp: "Formatted at 2026-11-20 15:00:57 on dist-test-slave-m239"
I20261120 15:00:57.599646 7934 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:57.607743 7934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:57.608208 7934 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:57.608377 7934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:57.608866 7934 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:00:57.609768 7934 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:00:57.609840 7934 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.609913 7934 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:00:57.609967 7934 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:57.619771 7934 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:35401
I20261120 15:00:57.619791 8061 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:35401 every 8 connection(s)
I20261120 15:00:57.620747 7934 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
I20261120 15:00:57.623824 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 7934
I20261120 15:00:57.623939 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1795186845751551-6849-0/minicluster-data/ts-3/wal/instance
I20261120 15:00:57.628548 8062 heartbeater.cc:345] Connected to a master server at 127.6.176.126:34189
I20261120 15:00:57.628773 8062 heartbeater.cc:462] Registering TS with master...
I20261120 15:00:57.629211 8062 heartbeater.cc:508] Master 127.6.176.126:34189 requested a full tablet report, sending...
I20261120 15:00:57.630213 7481 ts_manager.cc:210] Registered new tserver with Master: 8d501f0f24c04b78a19107f50163c497 (127.6.176.68:35401)
I20261120 15:00:57.630759 7481 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:52321
I20261120 15:00:57.638150 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:00:57.654137 7481 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46344:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261120 15:00:57.671384 7996 tablet_service.cc:1495] Processing CreateTablet for tablet f8fa795dca85434e843f604b667c63cd (DEFAULT_TABLE table=test-workload [id=4d34cdf32d254ace8a97d523c733438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:57.671404 7865 tablet_service.cc:1495] Processing CreateTablet for tablet f8fa795dca85434e843f604b667c63cd (DEFAULT_TABLE table=test-workload [id=4d34cdf32d254ace8a97d523c733438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:57.671415 7603 tablet_service.cc:1495] Processing CreateTablet for tablet f8fa795dca85434e843f604b667c63cd (DEFAULT_TABLE table=test-workload [id=4d34cdf32d254ace8a97d523c733438b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:00:57.672359 7603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fa795dca85434e843f604b667c63cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:57.672379 7865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fa795dca85434e843f604b667c63cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:57.672379 7996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fa795dca85434e843f604b667c63cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:57.677397 8084 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: Bootstrap starting.
I20261120 15:00:57.678303 8086 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Bootstrap starting.
I20261120 15:00:57.678706 8085 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Bootstrap starting.
I20261120 15:00:57.680078 8084 tablet_bootstrap.cc:654] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:57.680987 8085 tablet_bootstrap.cc:654] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:57.681027 8084 log.cc:826] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:57.681425 8086 tablet_bootstrap.cc:654] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:57.681931 8085 log.cc:826] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:57.682217 8086 log.cc:826] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:57.682706 8084 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: No bootstrap required, opened a new log
I20261120 15:00:57.682914 8084 ts_tablet_manager.cc:1449] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261120 15:00:57.683364 8085 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: No bootstrap required, opened a new log
I20261120 15:00:57.683565 8085 ts_tablet_manager.cc:1449] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20261120 15:00:57.683621 8086 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: No bootstrap required, opened a new log
I20261120 15:00:57.683840 8086 ts_tablet_manager.cc:1449] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261120 15:00:57.685537 8084 raft_consensus.cc:366] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.685787 8084 raft_consensus.cc:392] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:57.685861 8084 raft_consensus.cc:787] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a689928847f64cad850b8da6f8f2dafd, State: Initialized, Role: FOLLOWER
I20261120 15:00:57.686285 8084 consensus_queue.cc:261] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.686638 8085 raft_consensus.cc:366] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.686746 8086 raft_consensus.cc:366] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.686900 8086 raft_consensus.cc:392] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:57.686986 7669 heartbeater.cc:500] Master 127.6.176.126:34189 was elected leader, sending a full tablet report...
I20261120 15:00:57.686990 8086 raft_consensus.cc:787] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571f2655dabd4acb88f01d4f51af21dd, State: Initialized, Role: FOLLOWER
I20261120 15:00:57.687163 8084 ts_tablet_manager.cc:1480] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20261120 15:00:57.687311 8085 raft_consensus.cc:392] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:57.687431 8086 consensus_queue.cc:261] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.687572 8085 raft_consensus.cc:787] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d501f0f24c04b78a19107f50163c497, State: Initialized, Role: FOLLOWER
I20261120 15:00:57.688089 7931 heartbeater.cc:500] Master 127.6.176.126:34189 was elected leader, sending a full tablet report...
I20261120 15:00:57.688030 8085 consensus_queue.cc:261] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.688794 8085 ts_tablet_manager.cc:1480] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:00:57.689087 8062 heartbeater.cc:500] Master 127.6.176.126:34189 was elected leader, sending a full tablet report...
I20261120 15:00:57.689345 8086 ts_tablet_manager.cc:1480] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20261120 15:00:57.707549 7932 tablet.cc:2554] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 15:00:57.835438 8092 raft_consensus.cc:534] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:00:57.835615 8092 raft_consensus.cc:556] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.836843 8092 leader_election.cc:290] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571f2655dabd4acb88f01d4f51af21dd (127.6.176.67:41071), a689928847f64cad850b8da6f8f2dafd (127.6.176.65:34701)
I20261120 15:00:57.840942 7623 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "f8fa795dca85434e843f604b667c63cd" candidate_uuid: "8d501f0f24c04b78a19107f50163c497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a689928847f64cad850b8da6f8f2dafd" is_pre_election: true
I20261120 15:00:57.841267 7623 raft_consensus.cc:2062] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 8d501f0f24c04b78a19107f50163c497 in term 0
I20261120 15:00:57.841300 7885 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "f8fa795dca85434e843f604b667c63cd" candidate_uuid: "8d501f0f24c04b78a19107f50163c497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571f2655dabd4acb88f01d4f51af21dd" is_pre_election: true
I20261120 15:00:57.841584 7885 raft_consensus.cc:2062] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 8d501f0f24c04b78a19107f50163c497 in term 0
I20261120 15:00:57.841759 7949 leader_election.cc:304] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d501f0f24c04b78a19107f50163c497, a689928847f64cad850b8da6f8f2dafd; no voters:
I20261120 15:00:57.841974 8092 raft_consensus.cc:3024] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 15:00:57.842064 8092 raft_consensus.cc:534] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:00:57.842116 8092 raft_consensus.cc:3296] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:57.842410 8092 raft_consensus.cc:556] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.843590 8092 leader_election.cc:290] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [CANDIDATE]: Term 1 election: Requested vote from peers 571f2655dabd4acb88f01d4f51af21dd (127.6.176.67:41071), a689928847f64cad850b8da6f8f2dafd (127.6.176.65:34701)
I20261120 15:00:57.843889 7885 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "f8fa795dca85434e843f604b667c63cd" candidate_uuid: "8d501f0f24c04b78a19107f50163c497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571f2655dabd4acb88f01d4f51af21dd"
I20261120 15:00:57.843919 7623 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "f8fa795dca85434e843f604b667c63cd" candidate_uuid: "8d501f0f24c04b78a19107f50163c497" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a689928847f64cad850b8da6f8f2dafd"
I20261120 15:00:57.844020 7885 raft_consensus.cc:3296] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:57.844079 7623 raft_consensus.cc:3296] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:57.844290 7885 raft_consensus.cc:2062] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 8d501f0f24c04b78a19107f50163c497 in term 1
I20261120 15:00:57.844347 7623 raft_consensus.cc:2062] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 8d501f0f24c04b78a19107f50163c497 in term 1
I20261120 15:00:57.845643 7952 leader_election.cc:304] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571f2655dabd4acb88f01d4f51af21dd, 8d501f0f24c04b78a19107f50163c497; no voters:
I20261120 15:00:57.845834 8092 raft_consensus.cc:3024] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:00:57.845924 8092 raft_consensus.cc:744] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 1 LEADER]: Becoming Leader. State: Replica: 8d501f0f24c04b78a19107f50163c497, State: Running, Role: LEADER
I20261120 15:00:57.846273 8092 consensus_queue.cc:238] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } }
I20261120 15:00:57.849536 8092 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:00:57.849967 7479 catalog_manager.cc:5856] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d501f0f24c04b78a19107f50163c497 (127.6.176.68). New cstate: current_term: 1 leader_uuid: "8d501f0f24c04b78a19107f50163c497" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } health_report { overall_health: HEALTHY } } }
I20261120 15:00:57.852128 8096 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:00:57.912668 8016 consensus_queue.cc:238] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } }
I20261120 15:00:57.914160 7885 raft_consensus.cc:1338] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 1 FOLLOWER]: Refusing update from remote peer 8d501f0f24c04b78a19107f50163c497: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261120 15:00:57.914448 7623 raft_consensus.cc:1338] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 1 FOLLOWER]: Refusing update from remote peer 8d501f0f24c04b78a19107f50163c497: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261120 15:00:57.914613 8096 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 15:00:57.915004 8092 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 15:00:57.916863 8096 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7346930d1d97432a9707c73df3aff288 (127.6.176.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } } }
I20261120 15:00:57.917285 7885 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7346930d1d97432a9707c73df3aff288 (127.6.176.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } } }
I20261120 15:00:57.917447 7623 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7346930d1d97432a9707c73df3aff288 (127.6.176.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } } }
I20261120 15:00:57.920598 7481 catalog_manager.cc:5856] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd reported cstate change: config changed from index -1 to 2, NON_VOTER 7346930d1d97432a9707c73df3aff288 (127.6.176.66) added. New cstate: current_term: 1 leader_uuid: "8d501f0f24c04b78a19107f50163c497" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } } }
W20261120 15:00:57.920794 7949 consensus_peers.cc:597] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 -> Peer 7346930d1d97432a9707c73df3aff288 (127.6.176.66:46329): Couldn't send request to peer 7346930d1d97432a9707c73df3aff288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8fa795dca85434e843f604b667c63cd. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:00:58.301944 7800 heartbeater.cc:500] Master 127.6.176.126:34189 was elected leader, sending a full tablet report...
I20261120 15:00:58.393757 8108 ts_tablet_manager.cc:979] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Initiating tablet copy from peer 8d501f0f24c04b78a19107f50163c497 (127.6.176.68:35401)
I20261120 15:00:58.394301 8108 tablet_copy_client.cc:323] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: tablet copy: Beginning tablet copy session from remote peer at address 127.6.176.68:35401
I20261120 15:00:58.398057 8036 tablet_copy_service.cc:140] P 8d501f0f24c04b78a19107f50163c497: Received BeginTabletCopySession request for tablet f8fa795dca85434e843f604b667c63cd from peer 7346930d1d97432a9707c73df3aff288 ({username='slave'} at 127.6.176.66:51727)
I20261120 15:00:58.398221 8036 tablet_copy_service.cc:161] P 8d501f0f24c04b78a19107f50163c497: Beginning new tablet copy session on tablet f8fa795dca85434e843f604b667c63cd from peer 7346930d1d97432a9707c73df3aff288 at {username='slave'} at 127.6.176.66:51727: session id = 7346930d1d97432a9707c73df3aff288-f8fa795dca85434e843f604b667c63cd
I20261120 15:00:58.399528 8036 tablet_copy_source_session.cc:215] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: Tablet Copy: opened 0 blocks and 1 log segments
I20261120 15:00:58.403111 8108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8fa795dca85434e843f604b667c63cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:58.407976 8108 tablet_copy_client.cc:806] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: tablet copy: Starting download of 0 data blocks...
I20261120 15:00:58.408183 8108 tablet_copy_client.cc:670] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: tablet copy: Starting download of 1 WAL segments...
I20261120 15:00:58.409852 8108 tablet_copy_client.cc:538] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 15:00:58.411448 8108 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Bootstrap starting.
I20261120 15:00:58.415874 8108 log.cc:826] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:58.418092 8108 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 15:00:58.418417 8108 tablet_bootstrap.cc:492] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Bootstrap complete.
I20261120 15:00:58.418721 8108 ts_tablet_manager.cc:1449] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20261120 15:00:58.421666 8108 raft_consensus.cc:366] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } }
I20261120 15:00:58.421924 8108 raft_consensus.cc:787] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7346930d1d97432a9707c73df3aff288, State: Initialized, Role: LEARNER
I20261120 15:00:58.422260 8108 consensus_queue.cc:261] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: NON_VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: true } }
I20261120 15:00:58.422978 8108 ts_tablet_manager.cc:1480] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20261120 15:00:58.423787 8036 tablet_copy_service.cc:342] P 8d501f0f24c04b78a19107f50163c497: Request end of tablet copy session 7346930d1d97432a9707c73df3aff288-f8fa795dca85434e843f604b667c63cd received from {username='slave'} at 127.6.176.66:51727
I20261120 15:00:58.423894 8036 tablet_copy_service.cc:434] P 8d501f0f24c04b78a19107f50163c497: ending tablet copy session 7346930d1d97432a9707c73df3aff288-f8fa795dca85434e843f604b667c63cd on tablet f8fa795dca85434e843f604b667c63cd with peer 7346930d1d97432a9707c73df3aff288
I20261120 15:00:58.967780 7754 raft_consensus.cc:1279] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20261120 15:00:59.544781 8114 raft_consensus.cc:1122] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497: attempting to promote NON_VOTER 7346930d1d97432a9707c73df3aff288 to VOTER
I20261120 15:00:59.545301 8114 consensus_queue.cc:238] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } }
I20261120 15:00:59.546761 7623 raft_consensus.cc:1338] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 1 FOLLOWER]: Refusing update from remote peer 8d501f0f24c04b78a19107f50163c497: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261120 15:00:59.546757 7885 raft_consensus.cc:1338] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 1 FOLLOWER]: Refusing update from remote peer 8d501f0f24c04b78a19107f50163c497: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261120 15:00:59.546831 7754 raft_consensus.cc:1338] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [term 1 LEARNER]: Refusing update from remote peer 8d501f0f24c04b78a19107f50163c497: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261120 15:00:59.547245 8119 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 15:00:59.547428 8116 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 15:00:59.547551 8118 consensus_queue.cc:1049] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261120 15:00:59.549387 8116 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7346930d1d97432a9707c73df3aff288 (127.6.176.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } } }
I20261120 15:00:59.549816 7885 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P 571f2655dabd4acb88f01d4f51af21dd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7346930d1d97432a9707c73df3aff288 (127.6.176.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } } }
I20261120 15:00:59.551566 7480 catalog_manager.cc:5856] T f8fa795dca85434e843f604b667c63cd P 8d501f0f24c04b78a19107f50163c497 reported cstate change: config changed from index 2 to 3, 7346930d1d97432a9707c73df3aff288 (127.6.176.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d501f0f24c04b78a19107f50163c497" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20261120 15:00:59.553048 7623 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P a689928847f64cad850b8da6f8f2dafd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7346930d1d97432a9707c73df3aff288 (127.6.176.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } } }
I20261120 15:00:59.553129 7754 raft_consensus.cc:3183] T f8fa795dca85434e843f604b667c63cd P 7346930d1d97432a9707c73df3aff288 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7346930d1d97432a9707c73df3aff288 (127.6.176.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "571f2655dabd4acb88f01d4f51af21dd" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 41071 } } peers { permanent_uuid: "a689928847f64cad850b8da6f8f2dafd" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 34701 } } peers { permanent_uuid: "8d501f0f24c04b78a19107f50163c497" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 35401 } } peers { permanent_uuid: "7346930d1d97432a9707c73df3aff288" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 46329 } attrs { promote: false } } }
I20261120 15:00:59.605317 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7541
I20261120 15:00:59.611898 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7672
I20261120 15:00:59.617810 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7803
I20261120 15:00:59.623648 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7934
I20261120 15:00:59.629192 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 7450
2026-11-20T15:00:59Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestNonVoterPromotion (2973 ms)
[ RUN ] RaftConfigChangeITest.TestBulkChangeConfig
2026-11-20T15:00:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:00:59Z Disabled control of system clock
I20261120 15:00:59.660071 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:35571
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:35571
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:00:59.759778 8136 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:59.759951 8136 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:59.759989 8136 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:59.763329 8136 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:00:59.763407 8136 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:59.763428 8136 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:00:59.763446 8136 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:00:59.767262 8136 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:35571
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:59.768006 8136 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:59.768667 8136 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:59.773402 8141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:59.773429 8144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:59.773633 8142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:59.773995 8136 server_base.cc:1061] running on GCE node
I20261120 15:00:59.774434 8136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:59.775161 8136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:59.776317 8136 hybrid_clock.cc:648] HybridClock initialized: now 1795186859776300 us; error 29 us; skew 500 ppm
I20261120 15:00:59.777935 8136 webserver.cc:577] Webserver started at http://127.6.176.126:36029/ using document root <none> and password file <none>
I20261120 15:00:59.778481 8136 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:59.778553 8136 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:59.778777 8136 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:59.780090 8136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data/instance:
uuid: "68b41f8e3c0a499fac3a4c419536b220"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.780519 8136 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal/instance:
uuid: "68b41f8e3c0a499fac3a4c419536b220"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.783631 8136 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:00:59.785953 8150 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:59.787070 8136 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:00:59.787212 8136 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal
uuid: "68b41f8e3c0a499fac3a4c419536b220"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.787294 8136 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:00:59.800549 8136 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:00:59.800989 8136 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:00:59.801138 8136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:00:59.808117 8136 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:35571
I20261120 15:00:59.808118 8202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:35571 every 8 connection(s)
I20261120 15:00:59.809162 8136 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/data/info.pb
I20261120 15:00:59.812296 8203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:00:59.814688 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8136
I20261120 15:00:59.814831 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/master-0/wal/instance
I20261120 15:00:59.818121 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220: Bootstrap starting.
I20261120 15:00:59.820719 8203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220: Neither blocks nor log segments found. Creating new log.
I20261120 15:00:59.821532 8203 log.cc:826] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220: Log is configured to *not* fsync() on all Append() calls
I20261120 15:00:59.822911 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220: No bootstrap required, opened a new log
I20261120 15:00:59.825285 8203 raft_consensus.cc:366] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } }
I20261120 15:00:59.825470 8203 raft_consensus.cc:392] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:00:59.825551 8203 raft_consensus.cc:787] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b41f8e3c0a499fac3a4c419536b220, State: Initialized, Role: FOLLOWER
I20261120 15:00:59.825951 8203 consensus_queue.cc:261] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } }
I20261120 15:00:59.826128 8203 raft_consensus.cc:407] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:00:59.826220 8203 raft_consensus.cc:534] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:00:59.826320 8203 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:00:59.826637 8203 raft_consensus.cc:556] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } }
I20261120 15:00:59.827656 8203 leader_election.cc:304] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b41f8e3c0a499fac3a4c419536b220; no voters:
I20261120 15:00:59.827905 8203 leader_election.cc:290] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:00:59.827945 8208 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:00:59.828171 8208 raft_consensus.cc:744] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [term 1 LEADER]: Becoming Leader. State: Replica: 68b41f8e3c0a499fac3a4c419536b220, State: Running, Role: LEADER
I20261120 15:00:59.828549 8208 consensus_queue.cc:238] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } }
I20261120 15:00:59.829032 8203 sys_catalog.cc:565] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:00:59.830185 8210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b41f8e3c0a499fac3a4c419536b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } } }
I20261120 15:00:59.830302 8210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:59.830353 8209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b41f8e3c0a499fac3a4c419536b220. Latest consensus state: current_term: 1 leader_uuid: "68b41f8e3c0a499fac3a4c419536b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b41f8e3c0a499fac3a4c419536b220" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 35571 } } }
I20261120 15:00:59.830443 8209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220 [sys.catalog]: This master's current role is: LEADER
I20261120 15:00:59.831137 8217 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:00:59.833894 8217 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:00:59.839174 8217 catalog_manager.cc:1398] Generated new cluster ID: b2727c2b8e584836ac45532e47b7edbd
I20261120 15:00:59.839250 8217 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:00:59.848835 8217 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:00:59.849632 8217 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:00:59.854630 8217 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 68b41f8e3c0a499fac3a4c419536b220: Generated new TSK 0
I20261120 15:00:59.855193 8217 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:00:59.861445 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--builtin_ntp_servers=127.6.176.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:00:59.963974 8227 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261120 15:00:59.964206 8227 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:00:59.964272 8227 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:00:59.964359 8227 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:00:59.967878 8227 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:00:59.968039 8227 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:00:59.971942 8227 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--never_fsync=true
--heap_profile_path=/tmp/kudu.8227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:00:59.972832 8227 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:00:59.973605 8227 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:00:59.980134 8232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:59.980183 8227 server_base.cc:1061] running on GCE node
W20261120 15:00:59.980254 8233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:00:59.980129 8235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:00:59.980826 8227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:00:59.981518 8227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:00:59.982712 8227 hybrid_clock.cc:648] HybridClock initialized: now 1795186859982683 us; error 47 us; skew 500 ppm
I20261120 15:00:59.984700 8227 webserver.cc:577] Webserver started at http://127.6.176.65:41263/ using document root <none> and password file <none>
I20261120 15:00:59.985455 8227 fs_manager.cc:362] Metadata directory not provided
I20261120 15:00:59.985528 8227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:00:59.985786 8227 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:00:59.987627 8227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data/instance:
uuid: "7c9714e883184d0c8b09e3ed7a50d09e"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.988113 8227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance:
uuid: "7c9714e883184d0c8b09e3ed7a50d09e"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.992041 8227 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20261120 15:00:59.994678 8241 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:00:59.995915 8227 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:00:59.996027 8227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal
uuid: "7c9714e883184d0c8b09e3ed7a50d09e"
format_stamp: "Formatted at 2026-11-20 15:00:59 on dist-test-slave-m239"
I20261120 15:00:59.996120 8227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:00.017292 8227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:00.017846 8227 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:00.018018 8227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:00.018540 8227 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:00.019488 8227 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:00.019533 8227 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.019604 8227 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:00.019637 8227 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.028286 8227 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:35039
I20261120 15:01:00.028312 8354 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:35039 every 8 connection(s)
I20261120 15:01:00.029270 8227 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/data/info.pb
I20261120 15:01:00.036882 8355 heartbeater.cc:345] Connected to a master server at 127.6.176.126:35571
I20261120 15:01:00.037099 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8227
I20261120 15:01:00.037134 8355 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:00.037205 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-0/wal/instance
I20261120 15:01:00.037626 8355 heartbeater.cc:508] Master 127.6.176.126:35571 requested a full tablet report, sending...
I20261120 15:01:00.039199 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--builtin_ntp_servers=127.6.176.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261120 15:01:00.039184 8167 ts_manager.cc:210] Registered new tserver with Master: 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65:35039)
I20261120 15:01:00.040735 8167 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:55569
W20261120 15:01:00.140178 8358 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261120 15:01:00.140348 8358 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:00.140370 8358 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:00.140410 8358 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:00.143623 8358 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:00.143712 8358 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:01:00.147413 8358 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--never_fsync=true
--heap_profile_path=/tmp/kudu.8358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:00.148092 8358 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:00.148731 8358 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:00.154054 8366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.154084 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.154150 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:00.154309 8358 server_base.cc:1061] running on GCE node
I20261120 15:01:00.154675 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:00.155197 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:00.156339 8358 hybrid_clock.cc:648] HybridClock initialized: now 1795186860156319 us; error 43 us; skew 500 ppm
I20261120 15:01:00.157858 8358 webserver.cc:577] Webserver started at http://127.6.176.66:46407/ using document root <none> and password file <none>
I20261120 15:01:00.158406 8358 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:00.158453 8358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:00.158667 8358 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:00.160013 8358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data/instance:
uuid: "c1382ecb077e45539d13014453d00cae"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.160462 8358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance:
uuid: "c1382ecb077e45539d13014453d00cae"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.163602 8358 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:01:00.165773 8372 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.166697 8358 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:00.166815 8358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal
uuid: "c1382ecb077e45539d13014453d00cae"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.166905 8358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:00.175907 8358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:00.176352 8358 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:00.176508 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:00.176973 8358 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:00.177855 8358 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:00.177901 8358 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.177975 8358 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:00.178012 8358 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.186511 8358 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:37195
I20261120 15:01:00.186549 8485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:37195 every 8 connection(s)
I20261120 15:01:00.187482 8358 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/data/info.pb
I20261120 15:01:00.194412 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8358
I20261120 15:01:00.194542 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-1/wal/instance
I20261120 15:01:00.194774 8486 heartbeater.cc:345] Connected to a master server at 127.6.176.126:35571
I20261120 15:01:00.194984 8486 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:00.195518 8486 heartbeater.cc:508] Master 127.6.176.126:35571 requested a full tablet report, sending...
I20261120 15:01:00.196468 8167 ts_manager.cc:210] Registered new tserver with Master: c1382ecb077e45539d13014453d00cae (127.6.176.66:37195)
I20261120 15:01:00.196841 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--builtin_ntp_servers=127.6.176.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261120 15:01:00.197189 8167 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:44547
W20261120 15:01:00.297079 8489 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261120 15:01:00.297250 8489 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:00.297300 8489 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:00.297386 8489 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:00.300774 8489 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:00.300873 8489 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:00.304718 8489 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--never_fsync=true
--heap_profile_path=/tmp/kudu.8489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:00.305507 8489 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:00.306207 8489 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:00.311709 8497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.311736 8495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.311745 8494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:00.311970 8489 server_base.cc:1061] running on GCE node
I20261120 15:01:00.312376 8489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:00.312913 8489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:00.314090 8489 hybrid_clock.cc:648] HybridClock initialized: now 1795186860314062 us; error 39 us; skew 500 ppm
I20261120 15:01:00.315732 8489 webserver.cc:577] Webserver started at http://127.6.176.67:33977/ using document root <none> and password file <none>
I20261120 15:01:00.316334 8489 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:00.316385 8489 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:00.316618 8489 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:00.317902 8489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data/instance:
uuid: "054a5baea3b34a43a01e8dde873f3df8"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.318351 8489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance:
uuid: "054a5baea3b34a43a01e8dde873f3df8"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.321666 8489 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20261120 15:01:00.323899 8503 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.324867 8489 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:00.324994 8489 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal
uuid: "054a5baea3b34a43a01e8dde873f3df8"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.325101 8489 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:00.340724 8489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:00.341235 8489 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:00.341446 8489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:00.342005 8489 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:00.342941 8489 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:00.343008 8489 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.343089 8489 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:00.343137 8489 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.352084 8489 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:45647
I20261120 15:01:00.352130 8616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:45647 every 8 connection(s)
I20261120 15:01:00.353212 8489 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/data/info.pb
I20261120 15:01:00.361366 8617 heartbeater.cc:345] Connected to a master server at 127.6.176.126:35571
I20261120 15:01:00.361630 8617 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:00.362143 8617 heartbeater.cc:508] Master 127.6.176.126:35571 requested a full tablet report, sending...
I20261120 15:01:00.362417 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8489
I20261120 15:01:00.362530 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-2/wal/instance
I20261120 15:01:00.363294 8167 ts_manager.cc:210] Registered new tserver with Master: 054a5baea3b34a43a01e8dde873f3df8 (127.6.176.67:45647)
I20261120 15:01:00.363981 8167 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:37039
I20261120 15:01:00.364718 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:0
--local_ip_for_outbound_sockets=127.6.176.68
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--builtin_ntp_servers=127.6.176.84:45677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261120 15:01:00.466830 8620 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261120 15:01:00.467005 8620 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:00.467026 8620 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:00.467111 8620 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:00.470911 8620 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:00.471047 8620 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:01:00.474934 8620 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:45677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:35571
--never_fsync=true
--heap_profile_path=/tmp/kudu.8620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:00.475801 8620 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:00.476518 8620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:00.482034 8625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.482105 8626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.482154 8628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:00.482427 8620 server_base.cc:1061] running on GCE node
I20261120 15:01:00.482802 8620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:00.483426 8620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:00.484608 8620 hybrid_clock.cc:648] HybridClock initialized: now 1795186860484590 us; error 30 us; skew 500 ppm
I20261120 15:01:00.486316 8620 webserver.cc:577] Webserver started at http://127.6.176.68:46393/ using document root <none> and password file <none>
I20261120 15:01:00.486955 8620 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:00.487006 8620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:00.487260 8620 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:00.488631 8620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data/instance:
uuid: "24e83746f27e4dc3b70d5f736c875a70"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.489068 8620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal/instance:
uuid: "24e83746f27e4dc3b70d5f736c875a70"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.492549 8620 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:01:00.494863 8634 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.495882 8620 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:00.496013 8620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal
uuid: "24e83746f27e4dc3b70d5f736c875a70"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.496165 8620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:00.516201 8620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:00.516726 8620 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:00.516898 8620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:00.517426 8620 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:00.518390 8620 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:00.518458 8620 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.518529 8620 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:00.518575 8620 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.527206 8620 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:40675
I20261120 15:01:00.527240 8747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:40675 every 8 connection(s)
I20261120 15:01:00.528378 8620 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/data/info.pb
I20261120 15:01:00.530243 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8620
I20261120 15:01:00.530387 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1795186845751551-6849-0/minicluster-data/ts-3/wal/instance
I20261120 15:01:00.536267 8748 heartbeater.cc:345] Connected to a master server at 127.6.176.126:35571
I20261120 15:01:00.536504 8748 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:00.536985 8748 heartbeater.cc:508] Master 127.6.176.126:35571 requested a full tablet report, sending...
I20261120 15:01:00.537933 8167 ts_manager.cc:210] Registered new tserver with Master: 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675)
I20261120 15:01:00.538452 8167 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:57005
I20261120 15:01:00.544215 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:00.559556 8167 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50948:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261120 15:01:00.575868 8420 tablet_service.cc:1495] Processing CreateTablet for tablet 44945d5db52a446792fc1eef880441ed (DEFAULT_TABLE table=test-workload [id=2af95bbdb2ad42dbb02e90740ea38804]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:00.576416 8289 tablet_service.cc:1495] Processing CreateTablet for tablet 44945d5db52a446792fc1eef880441ed (DEFAULT_TABLE table=test-workload [id=2af95bbdb2ad42dbb02e90740ea38804]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:00.576893 8420 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44945d5db52a446792fc1eef880441ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:00.576974 8551 tablet_service.cc:1495] Processing CreateTablet for tablet 44945d5db52a446792fc1eef880441ed (DEFAULT_TABLE table=test-workload [id=2af95bbdb2ad42dbb02e90740ea38804]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:00.577278 8289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44945d5db52a446792fc1eef880441ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:00.577684 8551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44945d5db52a446792fc1eef880441ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:00.582453 8770 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Bootstrap starting.
I20261120 15:01:00.583374 8771 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: Bootstrap starting.
I20261120 15:01:00.583478 8772 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: Bootstrap starting.
I20261120 15:01:00.585238 8771 tablet_bootstrap.cc:654] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:00.585502 8770 tablet_bootstrap.cc:654] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:00.585502 8772 tablet_bootstrap.cc:654] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:00.586040 8771 log.cc:826] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:00.586357 8772 log.cc:826] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:00.586357 8770 log.cc:826] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:00.587555 8771 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: No bootstrap required, opened a new log
I20261120 15:01:00.587733 8772 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: No bootstrap required, opened a new log
I20261120 15:01:00.587766 8771 ts_tablet_manager.cc:1449] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20261120 15:01:00.587822 8770 tablet_bootstrap.cc:492] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: No bootstrap required, opened a new log
I20261120 15:01:00.587934 8772 ts_tablet_manager.cc:1449] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:00.587997 8770 ts_tablet_manager.cc:1449] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20261120 15:01:00.590494 8772 raft_consensus.cc:366] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.590694 8772 raft_consensus.cc:787] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1382ecb077e45539d13014453d00cae, State: Initialized, Role: FOLLOWER
I20261120 15:01:00.590859 8770 raft_consensus.cc:366] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.590862 8771 raft_consensus.cc:366] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.591056 8770 raft_consensus.cc:787] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054a5baea3b34a43a01e8dde873f3df8, State: Initialized, Role: FOLLOWER
I20261120 15:01:00.591050 8771 raft_consensus.cc:787] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9714e883184d0c8b09e3ed7a50d09e, State: Initialized, Role: FOLLOWER
I20261120 15:01:00.591109 8772 consensus_queue.cc:261] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.591406 8771 consensus_queue.cc:261] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.591406 8770 consensus_queue.cc:261] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.591861 8486 heartbeater.cc:500] Master 127.6.176.126:35571 was elected leader, sending a full tablet report...
I20261120 15:01:00.592177 8770 ts_tablet_manager.cc:1480] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:00.592176 8771 ts_tablet_manager.cc:1480] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:00.592525 8355 heartbeater.cc:500] Master 127.6.176.126:35571 was elected leader, sending a full tablet report...
I20261120 15:01:00.592975 8617 heartbeater.cc:500] Master 127.6.176.126:35571 was elected leader, sending a full tablet report...
I20261120 15:01:00.593146 8772 ts_tablet_manager.cc:1480] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20261120 15:01:00.604055 8618 tablet.cc:2554] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 15:01:00.622402 8571 tablet_service.cc:2028] Received Run Leader Election RPC: tablet_id: "44945d5db52a446792fc1eef880441ed"
dest_uuid: "054a5baea3b34a43a01e8dde873f3df8"
from {username='slave'} at 127.0.0.1:35850
I20261120 15:01:00.622644 8571 raft_consensus.cc:534] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20261120 15:01:00.622766 8571 raft_consensus.cc:3296] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:00.623068 8571 raft_consensus.cc:556] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.624769 8571 leader_election.cc:290] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [CANDIDATE]: Term 1 election: Requested vote from peers 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65:35039), c1382ecb077e45539d13014453d00cae (127.6.176.66:37195)
I20261120 15:01:00.629108 8309 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "44945d5db52a446792fc1eef880441ed" candidate_uuid: "054a5baea3b34a43a01e8dde873f3df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c9714e883184d0c8b09e3ed7a50d09e"
I20261120 15:01:00.629345 8309 raft_consensus.cc:3296] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:00.629721 8309 raft_consensus.cc:2062] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 054a5baea3b34a43a01e8dde873f3df8 in term 1
I20261120 15:01:00.632069 8440 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "44945d5db52a446792fc1eef880441ed" candidate_uuid: "054a5baea3b34a43a01e8dde873f3df8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1382ecb077e45539d13014453d00cae"
I20261120 15:01:00.632264 8440 raft_consensus.cc:3296] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:00.632321 8505 leader_election.cc:304] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054a5baea3b34a43a01e8dde873f3df8, 7c9714e883184d0c8b09e3ed7a50d09e; no voters:
I20261120 15:01:00.632591 8440 raft_consensus.cc:2062] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 054a5baea3b34a43a01e8dde873f3df8 in term 1
I20261120 15:01:00.632596 8778 raft_consensus.cc:3024] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:00.632824 8778 raft_consensus.cc:744] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Becoming Leader. State: Replica: 054a5baea3b34a43a01e8dde873f3df8, State: Running, Role: LEADER
I20261120 15:01:00.633190 8778 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.636785 8166 catalog_manager.cc:5856] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 054a5baea3b34a43a01e8dde873f3df8 (127.6.176.67). New cstate: current_term: 1 leader_uuid: "054a5baea3b34a43a01e8dde873f3df8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } health_report { overall_health: HEALTHY } } }
I20261120 15:01:00.637418 8778 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:00.645373 8787 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20261120 15:01:00.701874 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8136
I20261120 15:01:00.709231 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.709306 6849 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.712160 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.713722 8309 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261120 15:01:00.713804 8440 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261120 15:01:00.714213 8787 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20261120 15:01:00.714421 8785 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20261120 15:01:00.716799 8787 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.717131 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.719564 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.719614 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.719673 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } }
W20261120 15:01:00.719700 8506 consensus_peers.cc:597] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 -> Peer 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675): Couldn't send request to peer 24e83746f27e4dc3b70d5f736c875a70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44945d5db52a446792fc1eef880441ed. This is attempt 1: this message will repeat every 5th retry.
W20261120 15:01:00.720535 8617 heartbeater.cc:648] Failed to heartbeat to 127.6.176.126:35571 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.176.126:35571: connect: Connection refused (error 111)
I20261120 15:01:00.720561 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.720404 8440 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.721829 8309 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20261120 15:01:00.721901 8506 consensus_peers.cc:597] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 -> Peer 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675): Couldn't send request to peer 24e83746f27e4dc3b70d5f736c875a70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44945d5db52a446792fc1eef880441ed. This is attempt 1: this message will repeat every 5th retry.
W20261120 15:01:00.722052 8355 heartbeater.cc:648] Failed to heartbeat to 127.6.176.126:35571 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.176.126:35571: connect: Connection refused (error 111)
I20261120 15:01:00.722138 8439 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20261120 15:01:00.722502 8788 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20261120 15:01:00.722784 8778 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20261120 15:01:00.723277 8486 heartbeater.cc:648] Failed to heartbeat to 127.6.176.126:35571 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.176.126:35571: connect: Connection refused (error 111)
I20261120 15:01:00.724223 8785 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.724673 8439 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.724757 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.727277 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.727380 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } }
I20261120 15:01:00.728255 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } }
I20261120 15:01:00.729347 8309 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20261120 15:01:00.729470 8439 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20261120 15:01:00.729489 8506 consensus_peers.cc:597] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 -> Peer 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675): Couldn't send request to peer 24e83746f27e4dc3b70d5f736c875a70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44945d5db52a446792fc1eef880441ed. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:00.729734 8787 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20261120 15:01:00.729833 8785 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20261120 15:01:00.731132 8788 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } } }
I20261120 15:01:00.731562 8439 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } } }
I20261120 15:01:00.732232 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } } }
I20261120 15:01:00.733456 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.733494 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.733561 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.734661 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.734705 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: NON_VOTER last_known_addr { host: "127.6.176.67" port: 45647 } attrs { promote: false replace: false } }
I20261120 15:01:00.735792 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.735831 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: true replace: false } }
I20261120 15:01:00.736791 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.736829 6849 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.736897 6849 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.737492 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }
I20261120 15:01:00.738404 8439 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20261120 15:01:00.738723 8778 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20261120 15:01:00.740178 8785 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) evicted, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } }
I20261120 15:01:00.740624 8439 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) evicted, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } }
I20261120 15:01:00.742147 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.742218 6849 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.742277 6849 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.743162 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.743201 6849 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.743801 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.744725 8439 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20261120 15:01:00.744959 8506 consensus_peers.cc:597] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 -> Peer 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675): Couldn't send request to peer 24e83746f27e4dc3b70d5f736c875a70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44945d5db52a446792fc1eef880441ed. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:00.745026 8785 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20261120 15:01:00.746299 8778 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.746657 8439 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.748625 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.748674 6849 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.749287 8571 consensus_queue.cc:238] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.750382 8439 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20261120 15:01:00.750463 8309 raft_consensus.cc:1338] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Refusing update from remote peer 054a5baea3b34a43a01e8dde873f3df8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20261120 15:01:00.750725 8506 consensus_peers.cc:597] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 -> Peer 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68:40675): Couldn't send request to peer 24e83746f27e4dc3b70d5f736c875a70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44945d5db52a446792fc1eef880441ed. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:00.750839 8788 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20261120 15:01:00.750999 8785 consensus_queue.cc:1049] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20261120 15:01:00.752418 8309 raft_consensus.cc:918] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
W20261120 15:01:00.752477 8309 raft_consensus.cc:1647] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 14' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: true replace: false } } peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: NON_VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.752765 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) evicted, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } }
I20261120 15:01:00.754954 8309 raft_consensus.cc:1546] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261120 15:01:00.755237 8309 raft_consensus.cc:918] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }:
New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
W20261120 15:01:00.755283 8309 raft_consensus.cc:1647] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 15' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }:
New config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.755499 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 24e83746f27e4dc3b70d5f736c875a70 (127.6.176.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } }
I20261120 15:01:00.757329 8309 raft_consensus.cc:1546] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261120 15:01:00.758100 8785 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 054a5baea3b34a43a01e8dde873f3df8 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } } }
I20261120 15:01:00.758471 8309 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P 7c9714e883184d0c8b09e3ed7a50d09e [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } } }
I20261120 15:01:00.760080 8439 raft_consensus.cc:3183] T 44945d5db52a446792fc1eef880441ed P c1382ecb077e45539d13014453d00cae [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 7c9714e883184d0c8b09e3ed7a50d09e (127.6.176.65) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "c1382ecb077e45539d13014453d00cae" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 37195 } } peers { permanent_uuid: "054a5baea3b34a43a01e8dde873f3df8" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45647 } } peers { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } } peers { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } } }
I20261120 15:01:00.761479 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.761541 6849 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.761613 6849 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "7c9714e883184d0c8b09e3ed7a50d09e" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.6.176.65" port: 35039 } attrs { promote: false replace: false } }
I20261120 15:01:00.762583 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.762625 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: false } }
I20261120 15:01:00.763463 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.764259 6849 raft_config_change-itest.cc:349] submitting config change with changes:
I20261120 15:01:00.764317 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: false replace: true } }
I20261120 15:01:00.764377 6849 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "24e83746f27e4dc3b70d5f736c875a70" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 40675 } attrs { promote: true replace: false } }
I20261120 15:01:00.765937 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8227
I20261120 15:01:00.772702 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8358
I20261120 15:01:00.778609 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8489
I20261120 15:01:00.784659 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8620
2026-11-20T15:01:00Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestBulkChangeConfig (1153 ms)
[----------] 3 tests from RaftConfigChangeITest (15045 ms total)
[----------] 3 tests from SlowTabletBootstrapTest
[ RUN ] SlowTabletBootstrapTest.TestSlowBootstrap
I20261120 15:01:00.807857 6849 ts_itest-base.cc:115] Starting cluster with:
I20261120 15:01:00.807921 6849 ts_itest-base.cc:116] --------------
I20261120 15:01:00.807940 6849 ts_itest-base.cc:117] 4 tablet servers
I20261120 15:01:00.807972 6849 ts_itest-base.cc:118] 3 replicas per TS
I20261120 15:01:00.807997 6849 ts_itest-base.cc:119] --------------
2026-11-20T15:01:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:01:00Z Disabled control of system clock
I20261120 15:01:00.815765 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:38455
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:38455 with env {}
W20261120 15:01:00.916386 8818 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:00.916597 8818 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:00.916697 8818 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:00.920135 8818 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:01:00.920219 8818 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:00.920264 8818 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:01:00.920324 8818 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:01:00.924201 8818 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:38455
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:00.925019 8818 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:00.925737 8818 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:00.930708 8823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:00.930838 8818 server_base.cc:1061] running on GCE node
W20261120 15:01:00.930896 8826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:00.930685 8824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:00.931427 8818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:00.932121 8818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:00.933283 8818 hybrid_clock.cc:648] HybridClock initialized: now 1795186860933267 us; error 30 us; skew 500 ppm
I20261120 15:01:00.934835 8818 webserver.cc:577] Webserver started at http://127.6.176.126:36275/ using document root <none> and password file <none>
I20261120 15:01:00.935485 8818 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:00.935564 8818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:00.935806 8818 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:00.937088 8818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2352a644767845188291d7d1def05cfd"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.937543 8818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2352a644767845188291d7d1def05cfd"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.940771 8818 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:00.942926 8832 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:00.943908 8818 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:00.944062 8818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2352a644767845188291d7d1def05cfd"
format_stamp: "Formatted at 2026-11-20 15:01:00 on dist-test-slave-m239"
I20261120 15:01:00.944164 8818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:00.987597 8818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:00.988142 8818 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:00.988308 8818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:00.995183 8818 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:38455
I20261120 15:01:00.995199 8884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:38455 every 8 connection(s)
I20261120 15:01:00.996275 8818 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261120 15:01:00.999301 8885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:01.000847 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8818
I20261120 15:01:01.000996 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261120 15:01:01.004832 8885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd: Bootstrap starting.
I20261120 15:01:01.007335 8885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:01.008266 8885 log.cc:826] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:01.009667 8885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd: No bootstrap required, opened a new log
I20261120 15:01:01.012200 8885 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } }
I20261120 15:01:01.012367 8885 raft_consensus.cc:392] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:01.012437 8885 raft_consensus.cc:787] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2352a644767845188291d7d1def05cfd, State: Initialized, Role: FOLLOWER
I20261120 15:01:01.012825 8885 consensus_queue.cc:261] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } }
I20261120 15:01:01.012971 8885 raft_consensus.cc:407] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:01:01.013033 8885 raft_consensus.cc:534] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:01:01.013142 8885 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:01.013458 8885 raft_consensus.cc:556] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } }
I20261120 15:01:01.014385 8885 leader_election.cc:304] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2352a644767845188291d7d1def05cfd; no voters:
I20261120 15:01:01.014619 8885 leader_election.cc:290] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:01:01.014672 8890 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:01.014904 8890 raft_consensus.cc:744] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [term 1 LEADER]: Becoming Leader. State: Replica: 2352a644767845188291d7d1def05cfd, State: Running, Role: LEADER
I20261120 15:01:01.015226 8890 consensus_queue.cc:238] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } }
I20261120 15:01:01.015766 8885 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:01:01.016871 8892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2352a644767845188291d7d1def05cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } } }
I20261120 15:01:01.016987 8892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [sys.catalog]: This master's current role is: LEADER
I20261120 15:01:01.017014 8891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2352a644767845188291d7d1def05cfd. Latest consensus state: current_term: 1 leader_uuid: "2352a644767845188291d7d1def05cfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2352a644767845188291d7d1def05cfd" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38455 } } }
I20261120 15:01:01.017109 8891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd [sys.catalog]: This master's current role is: LEADER
I20261120 15:01:01.017364 8899 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:01:01.020748 8899 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:01:01.026156 8899 catalog_manager.cc:1398] Generated new cluster ID: d9b192f4c7234edba15da50fc6adf06e
I20261120 15:01:01.026228 8899 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:01:01.048149 8899 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:01:01.048995 8899 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:01:01.058655 8899 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 2352a644767845188291d7d1def05cfd: Generated new TSK 0
I20261120 15:01:01.059290 8899 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:01:01.062150 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261120 15:01:01.163997 8909 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:01.164214 8909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:01.164237 8909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:01.164326 8909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:01.167979 8909 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:01.168042 8909 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:01.168081 8909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:01.168205 8909 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:01.168242 8909 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:01:01.172441 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--never_fsync=true
--heap_profile_path=/tmp/kudu.8909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:01.173271 8909 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:01.174006 8909 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:01.179633 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:01.179808 8909 server_base.cc:1061] running on GCE node
W20261120 15:01:01.179632 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.179813 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:01.180334 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:01.180868 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:01.182029 8909 hybrid_clock.cc:648] HybridClock initialized: now 1795186861182017 us; error 29 us; skew 500 ppm
I20261120 15:01:01.183661 8909 webserver.cc:577] Webserver started at http://127.6.176.65:38457/ using document root <none> and password file <none>
I20261120 15:01:01.184242 8909 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:01.184314 8909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:01.184557 8909 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:01.185824 8909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "526c1d98d4bf4d31b897e3996fcb95b8"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.186270 8909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "526c1d98d4bf4d31b897e3996fcb95b8"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.189556 8909 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:01.191780 8923 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.192852 8909 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:01.192994 8909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "526c1d98d4bf4d31b897e3996fcb95b8"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.193101 8909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:01.205286 8909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:01.205710 8909 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:01.205870 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:01.206332 8909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:01.207538 8909 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:01.207600 8909 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.207669 8909 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:01.207700 8909 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.216290 8909 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:45223
I20261120 15:01:01.216315 9036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:45223 every 8 connection(s)
I20261120 15:01:01.217427 8909 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261120 15:01:01.224893 9037 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38455
I20261120 15:01:01.225134 9037 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:01.225598 9037 heartbeater.cc:508] Master 127.6.176.126:38455 requested a full tablet report, sending...
I20261120 15:01:01.227094 8849 ts_manager.cc:210] Registered new tserver with Master: 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65:45223)
I20261120 15:01:01.227612 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 8909
I20261120 15:01:01.227722 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261120 15:01:01.228525 8849 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:45911
I20261120 15:01:01.230023 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261120 15:01:01.331918 9040 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:01.332161 9040 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:01.332221 9040 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:01.332321 9040 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:01.335793 9040 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:01.335850 9040 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:01.335886 9040 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:01.335948 9040 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:01.335978 9040 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:01:01.339910 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--never_fsync=true
--heap_profile_path=/tmp/kudu.9040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:01.340674 9040 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:01.341338 9040 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:01.346839 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.346886 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.346927 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:01.347029 9040 server_base.cc:1061] running on GCE node
I20261120 15:01:01.347527 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:01.348076 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:01.349211 9040 hybrid_clock.cc:648] HybridClock initialized: now 1795186861349201 us; error 23 us; skew 500 ppm
I20261120 15:01:01.350752 9040 webserver.cc:577] Webserver started at http://127.6.176.66:42493/ using document root <none> and password file <none>
I20261120 15:01:01.351282 9040 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:01.351326 9040 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:01.351593 9040 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:01.352835 9040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "eb1b1b8565bc45629f72167450f7cdf5"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.353230 9040 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "eb1b1b8565bc45629f72167450f7cdf5"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.356380 9040 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20261120 15:01:01.358548 9054 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.359724 9040 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:01.359853 9040 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eb1b1b8565bc45629f72167450f7cdf5"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.359951 9040 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:01.378898 9040 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:01.379467 9040 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:01.379632 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:01.380249 9040 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:01.381188 9040 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:01.381235 9040 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.381276 9040 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:01.381297 9040 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.390228 9040 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:33623
I20261120 15:01:01.390257 9167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:33623 every 8 connection(s)
I20261120 15:01:01.391258 9040 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261120 15:01:01.395481 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9040
I20261120 15:01:01.395594 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261120 15:01:01.397913 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:01.401186 9168 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38455
I20261120 15:01:01.401408 9168 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:01.401896 9168 heartbeater.cc:508] Master 127.6.176.126:38455 requested a full tablet report, sending...
I20261120 15:01:01.402958 8848 ts_manager.cc:210] Registered new tserver with Master: eb1b1b8565bc45629f72167450f7cdf5 (127.6.176.66:33623)
I20261120 15:01:01.403579 8848 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:33351
W20261120 15:01:01.499709 9171 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:01.499948 9171 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:01.500020 9171 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:01.500118 9171 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:01.503489 9171 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:01.503574 9171 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:01.503647 9171 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:01.503787 9171 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:01.503857 9171 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:01.507930 9171 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--never_fsync=true
--heap_profile_path=/tmp/kudu.9171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:01.508773 9171 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:01.509562 9171 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:01.515192 9176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.515213 9177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.515267 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:01.515368 9171 server_base.cc:1061] running on GCE node
I20261120 15:01:01.515802 9171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:01.516386 9171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:01.517542 9171 hybrid_clock.cc:648] HybridClock initialized: now 1795186861517532 us; error 24 us; skew 500 ppm
I20261120 15:01:01.519156 9171 webserver.cc:577] Webserver started at http://127.6.176.67:35937/ using document root <none> and password file <none>
I20261120 15:01:01.519764 9171 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:01.519814 9171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:01.520056 9171 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:01.521337 9171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0c723135846d4473a9c22f4835966d3a"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.521767 9171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0c723135846d4473a9c22f4835966d3a"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.524969 9171 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:01.527271 9185 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.528363 9171 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:01.528496 9171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0c723135846d4473a9c22f4835966d3a"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.528612 9171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:01.541028 9171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:01.541451 9171 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:01.541617 9171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:01.542099 9171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:01.543051 9171 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:01.543119 9171 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.543191 9171 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:01.543232 9171 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.552004 9171 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:35245
I20261120 15:01:01.552043 9298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:35245 every 8 connection(s)
I20261120 15:01:01.552981 9171 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261120 15:01:01.553371 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9171
I20261120 15:01:01.553500 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261120 15:01:01.556154 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:0
--local_ip_for_outbound_sockets=127.6.176.68
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:01.561662 9299 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38455
I20261120 15:01:01.561910 9299 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:01.562541 9299 heartbeater.cc:508] Master 127.6.176.126:38455 requested a full tablet report, sending...
I20261120 15:01:01.563756 8848 ts_manager.cc:210] Registered new tserver with Master: 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:01.564330 8848 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:47061
W20261120 15:01:01.658566 9302 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:01.658766 9302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:01.658789 9302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:01.658892 9302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:01.662591 9302 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:01.662657 9302 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:01.662694 9302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:01.662839 9302 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:01.662880 9302 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:01:01.666992 9302 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38455
--never_fsync=true
--heap_profile_path=/tmp/kudu.9302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:01.667861 9302 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:01.668604 9302 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:01.674821 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.674844 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:01.674901 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:01.675103 9302 server_base.cc:1061] running on GCE node
I20261120 15:01:01.675475 9302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:01.675995 9302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:01.677148 9302 hybrid_clock.cc:648] HybridClock initialized: now 1795186861677124 us; error 36 us; skew 500 ppm
I20261120 15:01:01.678696 9302 webserver.cc:577] Webserver started at http://127.6.176.68:43493/ using document root <none> and password file <none>
I20261120 15:01:01.679244 9302 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:01.679291 9302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:01.679533 9302 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:01.680787 9302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3da744322e284671a3aeb5d51771dfed"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.681257 9302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3da744322e284671a3aeb5d51771dfed"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.684458 9302 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:01:01.686636 9316 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.687651 9302 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:01.687744 9302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3da744322e284671a3aeb5d51771dfed"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:01.687858 9302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:01.700098 9302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:01.700505 9302 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:01.700660 9302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:01.701157 9302 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:01.702104 9302 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:01.702149 9302 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.702217 9302 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:01.702255 9302 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:01.710942 9302 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:41271
I20261120 15:01:01.711011 9429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:41271 every 8 connection(s)
I20261120 15:01:01.711977 9302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261120 15:01:01.719590 9430 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38455
I20261120 15:01:01.719780 9430 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:01.720187 9430 heartbeater.cc:508] Master 127.6.176.126:38455 requested a full tablet report, sending...
I20261120 15:01:01.721133 8848 ts_manager.cc:210] Registered new tserver with Master: 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271)
I20261120 15:01:01.721736 8848 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:42967
I20261120 15:01:01.721779 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9302
I20261120 15:01:01.721891 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261120 15:01:01.724098 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:01.732203 8848 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57134:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261120 15:01:01.750980 8971 tablet_service.cc:1495] Processing CreateTablet for tablet a63c496b0c544619ae9668e6e4a54de2 (DEFAULT_TABLE table=TestTable [id=4d149aaaad4f4668838f47e55af35b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:01.750980 9364 tablet_service.cc:1495] Processing CreateTablet for tablet a63c496b0c544619ae9668e6e4a54de2 (DEFAULT_TABLE table=TestTable [id=4d149aaaad4f4668838f47e55af35b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:01.750941 9233 tablet_service.cc:1495] Processing CreateTablet for tablet a63c496b0c544619ae9668e6e4a54de2 (DEFAULT_TABLE table=TestTable [id=4d149aaaad4f4668838f47e55af35b52]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:01.751955 9364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a63c496b0c544619ae9668e6e4a54de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:01.751955 8971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a63c496b0c544619ae9668e6e4a54de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:01.751960 9233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a63c496b0c544619ae9668e6e4a54de2. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:01.757174 9447 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Bootstrap starting.
I20261120 15:01:01.758164 9448 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Bootstrap starting.
I20261120 15:01:01.759761 9447 tablet_bootstrap.cc:654] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:01.760457 9448 tablet_bootstrap.cc:654] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:01.760686 9447 log.cc:826] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:01.761377 9448 log.cc:826] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:01.761390 9449 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Bootstrap starting.
I20261120 15:01:01.762270 9447 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: No bootstrap required, opened a new log
I20261120 15:01:01.762488 9447 ts_tablet_manager.cc:1449] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:01.762877 9448 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: No bootstrap required, opened a new log
I20261120 15:01:01.763067 9448 ts_tablet_manager.cc:1449] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:01.763824 9449 tablet_bootstrap.cc:654] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:01.764659 9449 log.cc:826] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:01.765702 9447 raft_consensus.cc:366] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.765898 9447 raft_consensus.cc:392] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:01.765951 9447 raft_consensus.cc:787] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c723135846d4473a9c22f4835966d3a, State: Initialized, Role: FOLLOWER
I20261120 15:01:01.766134 9449 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: No bootstrap required, opened a new log
I20261120 15:01:01.766350 9449 ts_tablet_manager.cc:1449] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:01.766237 9448 raft_consensus.cc:366] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.766417 9448 raft_consensus.cc:392] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:01.766357 9447 consensus_queue.cc:261] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.766474 9448 raft_consensus.cc:787] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c1d98d4bf4d31b897e3996fcb95b8, State: Initialized, Role: FOLLOWER
I20261120 15:01:01.766887 9448 consensus_queue.cc:261] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.767033 9299 heartbeater.cc:500] Master 127.6.176.126:38455 was elected leader, sending a full tablet report...
I20261120 15:01:01.767593 9037 heartbeater.cc:500] Master 127.6.176.126:38455 was elected leader, sending a full tablet report...
I20261120 15:01:01.767738 9448 ts_tablet_manager.cc:1480] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261120 15:01:01.768450 9447 ts_tablet_manager.cc:1480] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20261120 15:01:01.769724 9449 raft_consensus.cc:366] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.769935 9449 raft_consensus.cc:392] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:01.769991 9449 raft_consensus.cc:787] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da744322e284671a3aeb5d51771dfed, State: Initialized, Role: FOLLOWER
I20261120 15:01:01.770572 9449 consensus_queue.cc:261] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.771396 9449 ts_tablet_manager.cc:1480] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20261120 15:01:01.771734 9430 heartbeater.cc:500] Master 127.6.176.126:38455 was elected leader, sending a full tablet report...
W20261120 15:01:01.812846 9431 tablet.cc:2554] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:01:01.819168 9038 tablet.cc:2554] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:01:01.854424 9300 tablet.cc:2554] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 15:01:01.972790 9453 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:01:01.973003 9453 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.974259 9453 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65:45223)
I20261120 15:01:01.978438 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "0c723135846d4473a9c22f4835966d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed" is_pre_election: true
I20261120 15:01:01.978750 9381 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 0c723135846d4473a9c22f4835966d3a in term 0
I20261120 15:01:01.978793 8991 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "0c723135846d4473a9c22f4835966d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" is_pre_election: true
I20261120 15:01:01.979130 8991 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 0c723135846d4473a9c22f4835966d3a in term 0
I20261120 15:01:01.979300 9189 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c723135846d4473a9c22f4835966d3a, 3da744322e284671a3aeb5d51771dfed; no voters:
I20261120 15:01:01.979570 9453 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 15:01:01.979661 9453 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:01:01.979724 9453 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:01.980127 9453 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.981585 9453 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [CANDIDATE]: Term 1 election: Requested vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65:45223)
I20261120 15:01:01.982170 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "0c723135846d4473a9c22f4835966d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed"
I20261120 15:01:01.982225 8991 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "0c723135846d4473a9c22f4835966d3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c1d98d4bf4d31b897e3996fcb95b8"
I20261120 15:01:01.982321 9381 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:01.982354 8991 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:01.982620 9381 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 0c723135846d4473a9c22f4835966d3a in term 1
I20261120 15:01:01.982631 8991 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 0c723135846d4473a9c22f4835966d3a in term 1
I20261120 15:01:01.983994 9189 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c723135846d4473a9c22f4835966d3a, 3da744322e284671a3aeb5d51771dfed; no voters:
I20261120 15:01:01.984243 9453 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:01.984356 9453 raft_consensus.cc:744] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 LEADER]: Becoming Leader. State: Replica: 0c723135846d4473a9c22f4835966d3a, State: Running, Role: LEADER
I20261120 15:01:01.984754 9453 consensus_queue.cc:238] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:01.987998 9459 consensus_queue.cc:1049] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:01.987879 8848 catalog_manager.cc:5856] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c723135846d4473a9c22f4835966d3a (127.6.176.67). New cstate: current_term: 1 leader_uuid: "0c723135846d4473a9c22f4835966d3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } health_report { overall_health: HEALTHY } } }
I20261120 15:01:01.988204 9460 consensus_queue.cc:1049] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:02.034970 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:02.036576 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 526c1d98d4bf4d31b897e3996fcb95b8 to finish bootstrapping
I20261120 15:01:02.046521 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0c723135846d4473a9c22f4835966d3a to finish bootstrapping
I20261120 15:01:02.051578 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3da744322e284671a3aeb5d51771dfed to finish bootstrapping
I20261120 15:01:02.056455 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9171
I20261120 15:01:02.063427 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:35245
--local_ip_for_outbound_sockets=127.6.176.67
--tserver_master_addrs=127.6.176.126:38455
--webserver_port=35937
--webserver_interface=127.6.176.67
--builtin_ntp_servers=127.6.176.84:34667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=4000 with env {}
W20261120 15:01:02.163228 9467 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:02.163450 9467 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:02.163512 9467 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:02.163617 9467 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=4000
W20261120 15:01:02.163658 9467 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:02.167073 9467 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:02.167130 9467 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:02.167164 9467 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:02.167307 9467 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:02.167390 9467 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:02.171245 9467 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:34667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:35245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=35937
--tserver_master_addrs=127.6.176.126:38455
--tablet_bootstrap_inject_latency_ms=4000
--never_fsync=true
--heap_profile_path=/tmp/kudu.9467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:02.172041 9467 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:02.172757 9467 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:02.178359 9473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:02.178431 9472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:02.178392 9475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:02.178571 9467 server_base.cc:1061] running on GCE node
I20261120 15:01:02.178900 9467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:02.179517 9467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:02.180662 9467 hybrid_clock.cc:648] HybridClock initialized: now 1795186862180637 us; error 40 us; skew 500 ppm
I20261120 15:01:02.182215 9467 webserver.cc:577] Webserver started at http://127.6.176.67:35937/ using document root <none> and password file <none>
I20261120 15:01:02.182745 9467 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:02.182826 9467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:02.185945 9467 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:02.187999 9481 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:02.188845 9467 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:02.188959 9467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0c723135846d4473a9c22f4835966d3a"
format_stamp: "Formatted at 2026-11-20 15:01:01 on dist-test-slave-m239"
I20261120 15:01:02.189302 9467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:02.201224 9467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:02.201639 9467 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:02.201790 9467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:02.202268 9467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:02.203399 9488 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261120 15:01:02.206753 9467 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 15:01:02.206820 9467 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20261120 15:01:02.206914 9467 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 15:01:02.208640 9467 ts_tablet_manager.cc:662] Registered 1 tablets
W20261120 15:01:02.208693 9488 ts_tablet_manager.cc:1428] Injecting 4000ms delay in tablet bootstrapping
I20261120 15:01:02.208698 9467 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20261120 15:01:02.217460 9467 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:35245
I20261120 15:01:02.218632 9467 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261120 15:01:02.219303 9595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:35245 every 8 connection(s)
I20261120 15:01:02.226641 9596 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38455
I20261120 15:01:02.226931 9596 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:02.227519 9596 heartbeater.cc:508] Master 127.6.176.126:38455 requested a full tablet report, sending...
I20261120 15:01:02.228631 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9467
I20261120 15:01:02.229111 8847 ts_manager.cc:210] Re-registered known tserver with Master: 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:02.229882 8847 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:44273
I20261120 15:01:02.405400 9168 heartbeater.cc:500] Master 127.6.176.126:38455 was elected leader, sending a full tablet report...
I20261120 15:01:03.231861 9596 heartbeater.cc:500] Master 127.6.176.126:38455 was elected leader, sending a full tablet report...
I20261120 15:01:03.492976 9599 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c723135846d4473a9c22f4835966d3a)
I20261120 15:01:03.493114 9599 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:03.493747 9600 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c723135846d4473a9c22f4835966d3a)
I20261120 15:01:03.493852 9600 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:03.494449 9599 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:03.494944 9600 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65:45223), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:03.499439 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed" is_pre_election: true
I20261120 15:01:03.499603 9381 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 526c1d98d4bf4d31b897e3996fcb95b8 in term 1
I20261120 15:01:03.500044 8991 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "3da744322e284671a3aeb5d51771dfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" is_pre_election: true
I20261120 15:01:03.500070 8927 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da744322e284671a3aeb5d51771dfed, 526c1d98d4bf4d31b897e3996fcb95b8; no voters:
I20261120 15:01:03.500192 8991 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 3da744322e284671a3aeb5d51771dfed in term 1
I20261120 15:01:03.500312 9599 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 15:01:03.500406 9599 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c723135846d4473a9c22f4835966d3a)
I20261120 15:01:03.500451 9599 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:01:03.500728 9599 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:03.500770 9320 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da744322e284671a3aeb5d51771dfed, 526c1d98d4bf4d31b897e3996fcb95b8; no voters:
I20261120 15:01:03.501000 9600 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 15:01:03.501065 9600 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c723135846d4473a9c22f4835966d3a)
I20261120 15:01:03.501168 9600 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:01:03.501415 9600 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:03.500792 9550 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "3da744322e284671a3aeb5d51771dfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a" is_pre_election: true
I20261120 15:01:03.500792 9549 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a" is_pre_election: true
I20261120 15:01:03.502230 9599 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 election: Requested vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:03.502573 9548 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a"
I20261120 15:01:03.502895 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed"
I20261120 15:01:03.503063 9381 raft_consensus.cc:2628] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 526c1d98d4bf4d31b897e3996fcb95b8 in current term 2: Already voted for candidate 3da744322e284671a3aeb5d51771dfed in this term.
I20261120 15:01:03.503101 9600 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [CANDIDATE]: Term 2 election: Requested vote from peers 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65:45223), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
W20261120 15:01:03.503180 8925 leader_election.cc:343] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:03.503435 9548 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "3da744322e284671a3aeb5d51771dfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a"
I20261120 15:01:03.503479 8991 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "3da744322e284671a3aeb5d51771dfed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "526c1d98d4bf4d31b897e3996fcb95b8"
W20261120 15:01:03.503526 8925 leader_election.cc:343] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:03.503575 8925 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 526c1d98d4bf4d31b897e3996fcb95b8; no voters: 0c723135846d4473a9c22f4835966d3a, 3da744322e284671a3aeb5d51771dfed
I20261120 15:01:03.503628 8991 raft_consensus.cc:2628] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3da744322e284671a3aeb5d51771dfed in current term 2: Already voted for candidate 526c1d98d4bf4d31b897e3996fcb95b8 in this term.
I20261120 15:01:03.503738 9599 raft_consensus.cc:2966] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20261120 15:01:03.503749 9318 leader_election.cc:343] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:03.503885 9320 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3da744322e284671a3aeb5d51771dfed; no voters: 0c723135846d4473a9c22f4835966d3a, 526c1d98d4bf4d31b897e3996fcb95b8
I20261120 15:01:03.504129 9610 raft_consensus.cc:2966] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261120 15:01:05.358256 9611 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:01:05.358383 9611 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:05.358829 9611 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:05.359251 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed" is_pre_election: true
I20261120 15:01:05.359258 9548 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a" is_pre_election: true
I20261120 15:01:05.359447 9381 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 526c1d98d4bf4d31b897e3996fcb95b8 in term 2
W20261120 15:01:05.359748 8925 leader_election.cc:343] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:05.359894 8927 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3da744322e284671a3aeb5d51771dfed, 526c1d98d4bf4d31b897e3996fcb95b8; no voters: 0c723135846d4473a9c22f4835966d3a
I20261120 15:01:05.360086 9611 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 15:01:05.360142 9611 raft_consensus.cc:534] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:01:05.360172 9611 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 2 FOLLOWER]: Advancing to term 3
I20261120 15:01:05.360394 9611 raft_consensus.cc:556] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:05.361523 9611 leader_election.cc:290] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 election: Requested vote from peers 3da744322e284671a3aeb5d51771dfed (127.6.176.68:41271), 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245)
I20261120 15:01:05.361901 9548 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0c723135846d4473a9c22f4835966d3a"
I20261120 15:01:05.361966 9381 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "a63c496b0c544619ae9668e6e4a54de2" candidate_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3da744322e284671a3aeb5d51771dfed"
I20261120 15:01:05.362100 9381 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 2 FOLLOWER]: Advancing to term 3
W20261120 15:01:05.362341 8925 leader_election.cc:343] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:05.362378 9381 raft_consensus.cc:2062] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate 526c1d98d4bf4d31b897e3996fcb95b8 in term 3
I20261120 15:01:05.363678 8927 leader_election.cc:304] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3da744322e284671a3aeb5d51771dfed, 526c1d98d4bf4d31b897e3996fcb95b8; no voters: 0c723135846d4473a9c22f4835966d3a
I20261120 15:01:05.363857 9611 raft_consensus.cc:3024] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 3 FOLLOWER]: Leader election won for term 3
I20261120 15:01:05.363988 9611 raft_consensus.cc:744] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [term 3 LEADER]: Becoming Leader. State: Replica: 526c1d98d4bf4d31b897e3996fcb95b8, State: Running, Role: LEADER
I20261120 15:01:05.364346 9611 consensus_queue.cc:238] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:05.366649 9381 raft_consensus.cc:1338] T a63c496b0c544619ae9668e6e4a54de2 P 3da744322e284671a3aeb5d51771dfed [term 3 FOLLOWER]: Refusing update from remote peer 526c1d98d4bf4d31b897e3996fcb95b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261120 15:01:05.367107 9611 consensus_queue.cc:1049] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 15:01:05.367192 8848 catalog_manager.cc:5856] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 reported cstate change: term changed from 1 to 3, leader changed from 0c723135846d4473a9c22f4835966d3a (127.6.176.67) to 526c1d98d4bf4d31b897e3996fcb95b8 (127.6.176.65). New cstate: current_term: 3 leader_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } health_report { overall_health: UNKNOWN } } }
W20261120 15:01:05.367635 8925 consensus_peers.cc:597] T a63c496b0c544619ae9668e6e4a54de2 P 526c1d98d4bf4d31b897e3996fcb95b8 -> Peer 0c723135846d4473a9c22f4835966d3a (127.6.176.67:35245): Couldn't send request to peer 0c723135846d4473a9c22f4835966d3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:06.209208 9488 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Bootstrap starting.
I20261120 15:01:06.216043 9488 log.cc:826] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:06.218880 9488 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 15:01:06.219238 9488 tablet_bootstrap.cc:492] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Bootstrap complete.
I20261120 15:01:06.219702 9488 ts_tablet_manager.cc:1449] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20261120 15:01:06.221967 9488 raft_consensus.cc:366] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:06.222282 9488 raft_consensus.cc:787] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c723135846d4473a9c22f4835966d3a, State: Initialized, Role: FOLLOWER
I20261120 15:01:06.222726 9488 consensus_queue.cc:261] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da744322e284671a3aeb5d51771dfed" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 41271 } } peers { permanent_uuid: "526c1d98d4bf4d31b897e3996fcb95b8" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 45223 } } peers { permanent_uuid: "0c723135846d4473a9c22f4835966d3a" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 35245 } }
I20261120 15:01:06.223486 9488 ts_tablet_manager.cc:1480] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:06.371109 9548 raft_consensus.cc:3296] T a63c496b0c544619ae9668e6e4a54de2 P 0c723135846d4473a9c22f4835966d3a [term 1 FOLLOWER]: Advancing to term 3
I20261120 15:01:09.239584 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8909
I20261120 15:01:09.245965 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9040
I20261120 15:01:09.251453 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9467
I20261120 15:01:09.257227 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9302
I20261120 15:01:09.262844 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 8818
2026-11-20T15:01:09Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestSlowBootstrap (8478 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehind
I20261120 15:01:09.286140 6849 ts_itest-base.cc:115] Starting cluster with:
I20261120 15:01:09.286185 6849 ts_itest-base.cc:116] --------------
I20261120 15:01:09.286198 6849 ts_itest-base.cc:117] 4 tablet servers
I20261120 15:01:09.286209 6849 ts_itest-base.cc:118] 3 replicas per TS
I20261120 15:01:09.286237 6849 ts_itest-base.cc:119] --------------
2026-11-20T15:01:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:01:09Z Disabled control of system clock
I20261120 15:01:09.294078 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:38127
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:38127 with env {}
W20261120 15:01:09.401418 9640 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:09.401650 9640 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:09.401759 9640 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:09.405198 9640 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:01:09.405294 9640 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:09.405352 9640 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:01:09.405418 9640 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:01:09.409363 9640 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:38127
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:38127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:09.410198 9640 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:09.410918 9640 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:09.416214 9646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.416255 9645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.416240 9648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:09.416424 9640 server_base.cc:1061] running on GCE node
I20261120 15:01:09.416891 9640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:09.417680 9640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:09.418859 9640 hybrid_clock.cc:648] HybridClock initialized: now 1795186869418828 us; error 45 us; skew 500 ppm
I20261120 15:01:09.420545 9640 webserver.cc:577] Webserver started at http://127.6.176.126:41915/ using document root <none> and password file <none>
I20261120 15:01:09.421140 9640 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:09.421216 9640 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:09.421476 9640 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:09.422767 9640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e5c6a4da4bea4ebca7c8a8256f163aee"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.423218 9640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e5c6a4da4bea4ebca7c8a8256f163aee"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.426385 9640 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20261120 15:01:09.428705 9654 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.429631 9640 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:01:09.429770 9640 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e5c6a4da4bea4ebca7c8a8256f163aee"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.429865 9640 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:09.443400 9640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:09.443943 9640 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:09.444133 9640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:09.451568 9640 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:38127
I20261120 15:01:09.451586 9706 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:38127 every 8 connection(s)
I20261120 15:01:09.452711 9640 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261120 15:01:09.453852 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9640
I20261120 15:01:09.453953 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261120 15:01:09.456154 9707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:09.461799 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee: Bootstrap starting.
I20261120 15:01:09.464146 9707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:09.464939 9707 log.cc:826] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:09.466459 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee: No bootstrap required, opened a new log
I20261120 15:01:09.469118 9707 raft_consensus.cc:366] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } }
I20261120 15:01:09.469293 9707 raft_consensus.cc:392] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:09.469363 9707 raft_consensus.cc:787] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c6a4da4bea4ebca7c8a8256f163aee, State: Initialized, Role: FOLLOWER
I20261120 15:01:09.469731 9707 consensus_queue.cc:261] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } }
I20261120 15:01:09.469858 9707 raft_consensus.cc:407] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:01:09.469944 9707 raft_consensus.cc:534] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:01:09.470049 9707 raft_consensus.cc:3296] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:09.470374 9707 raft_consensus.cc:556] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } }
I20261120 15:01:09.471683 9707 leader_election.cc:304] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c6a4da4bea4ebca7c8a8256f163aee; no voters:
I20261120 15:01:09.471910 9707 leader_election.cc:290] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:01:09.471974 9712 raft_consensus.cc:3024] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:09.472225 9712 raft_consensus.cc:744] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [term 1 LEADER]: Becoming Leader. State: Replica: e5c6a4da4bea4ebca7c8a8256f163aee, State: Running, Role: LEADER
I20261120 15:01:09.472594 9712 consensus_queue.cc:238] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } }
I20261120 15:01:09.473047 9707 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:01:09.474231 9713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c6a4da4bea4ebca7c8a8256f163aee. Latest consensus state: current_term: 1 leader_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } } }
I20261120 15:01:09.474771 9713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [sys.catalog]: This master's current role is: LEADER
I20261120 15:01:09.475035 9714 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c6a4da4bea4ebca7c8a8256f163aee" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 38127 } } }
I20261120 15:01:09.475124 9714 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee [sys.catalog]: This master's current role is: LEADER
I20261120 15:01:09.475311 9721 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:01:09.478281 9721 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:01:09.483558 9721 catalog_manager.cc:1398] Generated new cluster ID: fbe12d8f947840d0820ca34a976e16c1
I20261120 15:01:09.483629 9721 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:01:09.497857 9721 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:01:09.498976 9721 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:01:09.508479 9721 catalog_manager.cc:6237] T 00000000000000000000000000000000 P e5c6a4da4bea4ebca7c8a8256f163aee: Generated new TSK 0
I20261120 15:01:09.509194 9721 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:01:09.513496 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261120 15:01:09.614861 9731 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:09.615105 9731 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:09.615175 9731 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:09.615260 9731 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:09.618862 9731 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:09.618937 9731 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:09.619006 9731 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:09.619127 9731 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:09.619197 9731 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:01:09.623073 9731 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--never_fsync=true
--heap_profile_path=/tmp/kudu.9731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:09.623884 9731 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:09.624598 9731 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:09.630050 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.630095 9737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.630326 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:09.630674 9731 server_base.cc:1061] running on GCE node
I20261120 15:01:09.631022 9731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:09.631656 9731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:09.632828 9731 hybrid_clock.cc:648] HybridClock initialized: now 1795186869632804 us; error 41 us; skew 500 ppm
I20261120 15:01:09.634502 9731 webserver.cc:577] Webserver started at http://127.6.176.65:40785/ using document root <none> and password file <none>
I20261120 15:01:09.635099 9731 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:09.635174 9731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:09.635455 9731 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:09.636739 9731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a962c33db5c04a62863302aa9cd43092"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.637184 9731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a962c33db5c04a62863302aa9cd43092"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.640331 9731 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20261120 15:01:09.642467 9745 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.643446 9731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:09.643563 9731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a962c33db5c04a62863302aa9cd43092"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.643653 9731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:09.654991 9731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:09.655491 9731 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:09.655650 9731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:09.656111 9731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:09.656992 9731 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:09.657060 9731 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.657133 9731 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:09.657179 9731 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.665449 9731 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:44497
I20261120 15:01:09.665498 9858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:44497 every 8 connection(s)
I20261120 15:01:09.666402 9731 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261120 15:01:09.668835 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9731
I20261120 15:01:09.668932 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261120 15:01:09.671066 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:09.674538 9859 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38127
I20261120 15:01:09.674772 9859 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:09.675258 9859 heartbeater.cc:508] Master 127.6.176.126:38127 requested a full tablet report, sending...
I20261120 15:01:09.677057 9671 ts_manager.cc:210] Registered new tserver with Master: a962c33db5c04a62863302aa9cd43092 (127.6.176.65:44497)
I20261120 15:01:09.678419 9671 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:57915
W20261120 15:01:09.772100 9862 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:09.772307 9862 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:09.772328 9862 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:09.772368 9862 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:09.775810 9862 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:09.775861 9862 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:09.775894 9862 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:09.775954 9862 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:09.775978 9862 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:01:09.779810 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--never_fsync=true
--heap_profile_path=/tmp/kudu.9862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:09.780550 9862 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:09.781217 9862 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:09.786599 9867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.786669 9870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.786788 9868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:09.786945 9862 server_base.cc:1061] running on GCE node
I20261120 15:01:09.787496 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:09.788062 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:09.789244 9862 hybrid_clock.cc:648] HybridClock initialized: now 1795186869789215 us; error 41 us; skew 500 ppm
I20261120 15:01:09.791028 9862 webserver.cc:577] Webserver started at http://127.6.176.66:46205/ using document root <none> and password file <none>
I20261120 15:01:09.791734 9862 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:09.791810 9862 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:09.792037 9862 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:09.793323 9862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b3293a34669c4d5ba10faa4bb5340e61"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.793776 9862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b3293a34669c4d5ba10faa4bb5340e61"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.796976 9862 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20261120 15:01:09.799256 9876 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.800246 9862 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20261120 15:01:09.800375 9862 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b3293a34669c4d5ba10faa4bb5340e61"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.800472 9862 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:09.812312 9862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:09.812819 9862 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:09.812994 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:09.813481 9862 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:09.814417 9862 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:09.814488 9862 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.814563 9862 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:09.814616 9862 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.823320 9862 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:40691
I20261120 15:01:09.823397 9989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:40691 every 8 connection(s)
I20261120 15:01:09.824402 9862 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261120 15:01:09.826359 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9862
I20261120 15:01:09.826468 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261120 15:01:09.828804 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:09.832742 9990 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38127
I20261120 15:01:09.832967 9990 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:09.833485 9990 heartbeater.cc:508] Master 127.6.176.126:38127 requested a full tablet report, sending...
I20261120 15:01:09.834545 9671 ts_manager.cc:210] Registered new tserver with Master: b3293a34669c4d5ba10faa4bb5340e61 (127.6.176.66:40691)
I20261120 15:01:09.835124 9671 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:55367
W20261120 15:01:09.931488 9993 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:09.931795 9993 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:09.931859 9993 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:09.931955 9993 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:09.935606 9993 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:09.935695 9993 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:09.935770 9993 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:09.935911 9993 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:09.935984 9993 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:09.939930 9993 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--never_fsync=true
--heap_profile_path=/tmp/kudu.9993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:09.940763 9993 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:09.941476 9993 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:09.947058 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.947058 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:09.947140 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:09.947633 9993 server_base.cc:1061] running on GCE node
I20261120 15:01:09.947970 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:09.948535 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:09.949674 9993 hybrid_clock.cc:648] HybridClock initialized: now 1795186869949646 us; error 43 us; skew 500 ppm
I20261120 15:01:09.951247 9993 webserver.cc:577] Webserver started at http://127.6.176.67:38683/ using document root <none> and password file <none>
I20261120 15:01:09.951825 9993 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:09.951874 9993 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:09.952049 9993 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:09.953261 9993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.953647 9993 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.956759 9993 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:09.958799 10007 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.959753 9993 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:09.959878 9993 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf"
format_stamp: "Formatted at 2026-11-20 15:01:09 on dist-test-slave-m239"
I20261120 15:01:09.959972 9993 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:09.970106 9993 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:09.970525 9993 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:09.970681 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:09.971154 9993 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:09.972112 9993 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:09.972187 9993 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.972270 9993 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:09.972313 9993 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:09.980877 9993 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:45905
I20261120 15:01:09.980901 10120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:45905 every 8 connection(s)
I20261120 15:01:09.981807 9993 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261120 15:01:09.984323 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 9993
I20261120 15:01:09.984409 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261120 15:01:09.986564 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:0
--local_ip_for_outbound_sockets=127.6.176.68
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:09.990319 10121 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38127
I20261120 15:01:09.990559 10121 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:09.990995 10121 heartbeater.cc:508] Master 127.6.176.126:38127 requested a full tablet report, sending...
I20261120 15:01:09.992151 9671 ts_manager.cc:210] Registered new tserver with Master: 8b3a10a9b9ce4deb9d7fda3d9f04a5bf (127.6.176.67:45905)
I20261120 15:01:09.992981 9671 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:45745
W20261120 15:01:10.090852 10124 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:10.091049 10124 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:10.091073 10124 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:10.091111 10124 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:10.094370 10124 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:10.094421 10124 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:10.094456 10124 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:10.094518 10124 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:10.094547 10124 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:01:10.098272 10124 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:38127
--never_fsync=true
--heap_profile_path=/tmp/kudu.10124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:10.098965 10124 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:10.099668 10124 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:10.105203 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:10.105213 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:10.105268 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:10.105394 10124 server_base.cc:1061] running on GCE node
I20261120 15:01:10.105785 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:10.106305 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:10.107434 10124 hybrid_clock.cc:648] HybridClock initialized: now 1795186870107422 us; error 26 us; skew 500 ppm
I20261120 15:01:10.109005 10124 webserver.cc:577] Webserver started at http://127.6.176.68:38313/ using document root <none> and password file <none>
I20261120 15:01:10.109565 10124 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:10.109614 10124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:10.109828 10124 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:10.111059 10124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4bf3c4e5628248e085e90354d74818e2"
format_stamp: "Formatted at 2026-11-20 15:01:10 on dist-test-slave-m239"
I20261120 15:01:10.111547 10124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4bf3c4e5628248e085e90354d74818e2"
format_stamp: "Formatted at 2026-11-20 15:01:10 on dist-test-slave-m239"
I20261120 15:01:10.114573 10124 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20261120 15:01:10.116760 10138 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:10.117739 10124 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:01:10.117853 10124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4bf3c4e5628248e085e90354d74818e2"
format_stamp: "Formatted at 2026-11-20 15:01:10 on dist-test-slave-m239"
I20261120 15:01:10.117944 10124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:10.129860 10124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:10.130374 10124 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:10.130553 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:10.131053 10124 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:10.132071 10124 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:10.132117 10124 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:10.132186 10124 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:10.132221 10124 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:10.140512 10124 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:36605
I20261120 15:01:10.140553 10251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:36605 every 8 connection(s)
I20261120 15:01:10.141490 10124 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261120 15:01:10.142854 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10124
I20261120 15:01:10.142992 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261120 15:01:10.149744 10252 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38127
I20261120 15:01:10.149986 10252 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:10.150475 10252 heartbeater.cc:508] Master 127.6.176.126:38127 requested a full tablet report, sending...
I20261120 15:01:10.151489 9671 ts_manager.cc:210] Registered new tserver with Master: 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605)
I20261120 15:01:10.152029 9671 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:51341
I20261120 15:01:10.157132 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:10.165693 9671 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40022:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261120 15:01:10.184131 9924 tablet_service.cc:1495] Processing CreateTablet for tablet dbb500c360d4412d8029050e6ab674cc (DEFAULT_TABLE table=TestTable [id=0288dd7430374c0eb8d8af270ac6b8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:10.184131 10055 tablet_service.cc:1495] Processing CreateTablet for tablet dbb500c360d4412d8029050e6ab674cc (DEFAULT_TABLE table=TestTable [id=0288dd7430374c0eb8d8af270ac6b8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:10.184293 10186 tablet_service.cc:1495] Processing CreateTablet for tablet dbb500c360d4412d8029050e6ab674cc (DEFAULT_TABLE table=TestTable [id=0288dd7430374c0eb8d8af270ac6b8eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:10.185062 10186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb500c360d4412d8029050e6ab674cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:10.185088 10055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb500c360d4412d8029050e6ab674cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:10.185088 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb500c360d4412d8029050e6ab674cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:10.190155 10269 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Bootstrap starting.
I20261120 15:01:10.191092 10270 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Bootstrap starting.
I20261120 15:01:10.191663 10271 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Bootstrap starting.
I20261120 15:01:10.192754 10269 tablet_bootstrap.cc:654] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:10.193437 10270 tablet_bootstrap.cc:654] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:10.193512 10269 log.cc:826] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:10.193873 10271 tablet_bootstrap.cc:654] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:10.194312 10270 log.cc:826] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:10.194693 10271 log.cc:826] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:10.194952 10269 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: No bootstrap required, opened a new log
I20261120 15:01:10.195186 10269 ts_tablet_manager.cc:1449] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20261120 15:01:10.195760 10270 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: No bootstrap required, opened a new log
I20261120 15:01:10.195951 10270 ts_tablet_manager.cc:1449] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:10.196201 10271 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: No bootstrap required, opened a new log
I20261120 15:01:10.196404 10271 ts_tablet_manager.cc:1449] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:10.198028 10269 raft_consensus.cc:366] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.198218 10269 raft_consensus.cc:392] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:10.198272 10269 raft_consensus.cc:787] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf3c4e5628248e085e90354d74818e2, State: Initialized, Role: FOLLOWER
I20261120 15:01:10.198657 10269 consensus_queue.cc:261] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.198946 10270 raft_consensus.cc:366] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.199133 10270 raft_consensus.cc:392] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:10.199226 10270 raft_consensus.cc:787] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3293a34669c4d5ba10faa4bb5340e61, State: Initialized, Role: FOLLOWER
I20261120 15:01:10.199381 10252 heartbeater.cc:500] Master 127.6.176.126:38127 was elected leader, sending a full tablet report...
I20261120 15:01:10.199390 10271 raft_consensus.cc:366] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.199560 10271 raft_consensus.cc:392] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:10.199611 10271 raft_consensus.cc:787] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b3a10a9b9ce4deb9d7fda3d9f04a5bf, State: Initialized, Role: FOLLOWER
I20261120 15:01:10.199640 10270 consensus_queue.cc:261] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.199970 10271 consensus_queue.cc:261] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.200419 10270 ts_tablet_manager.cc:1480] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20261120 15:01:10.200511 9990 heartbeater.cc:500] Master 127.6.176.126:38127 was elected leader, sending a full tablet report...
I20261120 15:01:10.200680 10271 ts_tablet_manager.cc:1480] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:10.200950 10121 heartbeater.cc:500] Master 127.6.176.126:38127 was elected leader, sending a full tablet report...
I20261120 15:01:10.201593 10269 ts_tablet_manager.cc:1480] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20261120 15:01:10.225804 9991 tablet.cc:2554] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:01:10.242269 10253 tablet.cc:2554] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261120 15:01:10.283089 10122 tablet.cc:2554] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261120 15:01:10.476603 10277 raft_consensus.cc:534] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:01:10.476832 10277 raft_consensus.cc:556] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.478020 10277 leader_election.cc:290] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3293a34669c4d5ba10faa4bb5340e61 (127.6.176.66:40691), 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605)
I20261120 15:01:10.482498 9944 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "dbb500c360d4412d8029050e6ab674cc" candidate_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3293a34669c4d5ba10faa4bb5340e61" is_pre_election: true
I20261120 15:01:10.482547 10206 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "dbb500c360d4412d8029050e6ab674cc" candidate_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf3c4e5628248e085e90354d74818e2" is_pre_election: true
I20261120 15:01:10.482836 10206 raft_consensus.cc:2062] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 8b3a10a9b9ce4deb9d7fda3d9f04a5bf in term 0
I20261120 15:01:10.482836 9944 raft_consensus.cc:2062] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 8b3a10a9b9ce4deb9d7fda3d9f04a5bf in term 0
I20261120 15:01:10.483397 10011 leader_election.cc:304] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bf3c4e5628248e085e90354d74818e2, 8b3a10a9b9ce4deb9d7fda3d9f04a5bf, b3293a34669c4d5ba10faa4bb5340e61; no voters:
I20261120 15:01:10.483623 10277 raft_consensus.cc:3024] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 15:01:10.483686 10277 raft_consensus.cc:534] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:01:10.483735 10277 raft_consensus.cc:3296] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:10.484119 10277 raft_consensus.cc:556] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.485287 10277 leader_election.cc:290] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [CANDIDATE]: Term 1 election: Requested vote from peers b3293a34669c4d5ba10faa4bb5340e61 (127.6.176.66:40691), 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605)
I20261120 15:01:10.485641 9944 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "dbb500c360d4412d8029050e6ab674cc" candidate_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3293a34669c4d5ba10faa4bb5340e61"
I20261120 15:01:10.485678 10206 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "dbb500c360d4412d8029050e6ab674cc" candidate_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf3c4e5628248e085e90354d74818e2"
I20261120 15:01:10.485772 9944 raft_consensus.cc:3296] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:10.485838 10206 raft_consensus.cc:3296] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:10.486088 9944 raft_consensus.cc:2062] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 8b3a10a9b9ce4deb9d7fda3d9f04a5bf in term 1
I20261120 15:01:10.486166 10206 raft_consensus.cc:2062] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 8b3a10a9b9ce4deb9d7fda3d9f04a5bf in term 1
I20261120 15:01:10.487608 10011 leader_election.cc:304] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf3c4e5628248e085e90354d74818e2, 8b3a10a9b9ce4deb9d7fda3d9f04a5bf; no voters:
I20261120 15:01:10.487783 10277 raft_consensus.cc:3024] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:10.487926 10277 raft_consensus.cc:744] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 LEADER]: Becoming Leader. State: Replica: 8b3a10a9b9ce4deb9d7fda3d9f04a5bf, State: Running, Role: LEADER
I20261120 15:01:10.488317 10277 consensus_queue.cc:238] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:10.491528 10277 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:10.491647 10281 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:10.491742 9670 catalog_manager.cc:5856] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b3a10a9b9ce4deb9d7fda3d9f04a5bf (127.6.176.67). New cstate: current_term: 1 leader_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } health_report { overall_health: UNKNOWN } } }
I20261120 15:01:10.538650 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:10.540331 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b3293a34669c4d5ba10faa4bb5340e61 to finish bootstrapping
I20261120 15:01:10.545665 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8b3a10a9b9ce4deb9d7fda3d9f04a5bf to finish bootstrapping
I20261120 15:01:10.550635 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4bf3c4e5628248e085e90354d74818e2 to finish bootstrapping
I20261120 15:01:10.555137 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10124
I20261120 15:01:10.575435 6849 test_workload.cc:406] TestWorkload: Skipping table creation because table TestTable already exists
I20261120 15:01:10.575698 6849 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
W20261120 15:01:10.589648 10011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.176.68:36605: connect: Connection refused (error 111)
W20261120 15:01:10.590955 10011 consensus_peers.cc:597] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf -> Peer 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605): Couldn't send request to peer 4bf3c4e5628248e085e90354d74818e2. Status: Network error: Client connection negotiation failed: client connection to 127.6.176.68:36605: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:10.627795 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling FlushMRSOp(dbb500c360d4412d8029050e6ab674cc): 1709321 bytes log retention, and flush 2998537 bytes memory
I20261120 15:01:10.671635 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: FlushMRSOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.044s user 0.039s sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":6673,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":17,"thread_start_us":242,"threads_started":2}
I20261120 15:01:10.673347 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 1709321 bytes of WAL
I20261120 15:01:10.673640 9881 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 2 log segments from log reader
I20261120 15:01:10.673700 9881 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000001 (ops 1-7)
I20261120 15:01:10.673755 9881 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000002 (ops 8-14)
I20261120 15:01:10.674427 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:10.674695 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling FlushMRSOp(dbb500c360d4412d8029050e6ab674cc): 2366935 bytes log retention, and flush 2998537 bytes memory
I20261120 15:01:10.680529 9859 heartbeater.cc:500] Master 127.6.176.126:38127 was elected leader, sending a full tablet report...
I20261120 15:01:10.686584 10122 maintenance_manager.cc:419] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Scheduling FlushMRSOp(dbb500c360d4412d8029050e6ab674cc): 4602240 bytes log retention, and flush 6119689 bytes memory
I20261120 15:01:10.686998 6849 raft_consensus-itest-base.cc:274] Waiting for log GC on 8b3a10a9b9ce4deb9d7fda3d9f04a5bf
I20261120 15:01:10.697737 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: FlushMRSOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.023s user 0.020s sys 0.000s Metrics: {"bytes_written":135803,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":1582,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":630,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":14}
I20261120 15:01:10.698647 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 789029 bytes of WAL
I20261120 15:01:10.698827 9881 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 1 log segments from log reader
I20261120 15:01:10.698869 9881 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000003 (ops 15-20)
I20261120 15:01:10.699198 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:10.699488 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling CompactRowSetsOp(dbb500c360d4412d8029050e6ab674cc): perf score=0.962068
I20261120 15:01:10.747920 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: CompactRowSetsOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.048s user 0.048s sys 0.000s Metrics: {"bytes_written":135937,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264192,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":93,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":721,"lbm_reads_lt_1ms":54,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":31,"thread_start_us":319,"threads_started":5}
I20261120 15:01:10.748629 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc): 479 bytes on disk
I20261120 15:01:10.749228 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20261120 15:01:10.749867 10012 maintenance_manager.cc:643] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: FlushMRSOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.063s user 0.061s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":3510,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":202,"threads_started":2}
I20261120 15:01:10.750765 10122 maintenance_manager.cc:419] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 2761328 bytes of WAL
I20261120 15:01:10.751041 10012 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 3 log segments from log reader
I20261120 15:01:10.751099 10012 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000001 (ops 1-8)
I20261120 15:01:10.751150 10012 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000002 (ops 9-15)
I20261120 15:01:10.751180 10012 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000003 (ops 16-22)
I20261120 15:01:10.751984 10012 maintenance_manager.cc:643] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:10.752279 10122 maintenance_manager.cc:419] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Scheduling UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc): 549 bytes on disk
I20261120 15:01:10.752758 10012 maintenance_manager.cc:643] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20261120 15:01:10.758718 6849 raft_consensus-itest-base.cc:279] Log GC complete on 8b3a10a9b9ce4deb9d7fda3d9f04a5bf
I20261120 15:01:10.994450 10277 consensus_queue.cc:754] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: The logs necessary to catch up peer 4bf3c4e5628248e085e90354d74818e2 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 3..32: Segment 1 which contained index 3 has been GCed)
I20261120 15:01:10.994546 10277 consensus_queue.cc:580] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: The logs necessary to catch up peer 4bf3c4e5628248e085e90354d74818e2 have been garbage collected. The replica will never be able to catch up
I20261120 15:01:10.997591 10075 consensus_queue.cc:238] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 1.38, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } }
I20261120 15:01:10.998747 9943 raft_consensus.cc:1338] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Refusing update from remote peer 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 38. Preceding OpId from leader: term: 1 index: 39. (index mismatch)
I20261120 15:01:10.999179 10277 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20261120 15:01:11.000860 10281 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 LEADER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } }
I20261120 15:01:11.001367 9943 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Committing config change with OpId 1.39: config changed from index -1 to 39, VOTER 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } }
I20261120 15:01:11.001830 9657 catalog_manager.cc:5263] ChangeConfig:REMOVE_PEER RPC for tablet dbb500c360d4412d8029050e6ab674cc with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261120 15:01:11.002614 9671 catalog_manager.cc:5856] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf reported cstate change: config changed from index -1 to 39, VOTER 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68) evicted. New cstate: current_term: 1 leader_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } health_report { overall_health: HEALTHY } } }
W20261120 15:01:11.005700 9658 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.176.68:36605: connect: Connection refused (error 111)
I20261120 15:01:11.006475 10075 consensus_queue.cc:238] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 1.39, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } }
W20261120 15:01:11.006547 9658 catalog_manager.cc:4808] TS 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbb500c360d4412d8029050e6ab674cc: Network error: Client connection negotiation failed: client connection to 127.6.176.68:36605: connect: Connection refused (error 111)
I20261120 15:01:11.007997 9943 raft_consensus.cc:1338] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Refusing update from remote peer 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 39. Preceding OpId from leader: term: 1 index: 40. (index mismatch)
I20261120 15:01:11.008430 10281 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20261120 15:01:11.009898 10277 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 LEADER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER a962c33db5c04a62863302aa9cd43092 (127.6.176.65) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } } }
I20261120 15:01:11.010562 9657 catalog_manager.cc:5263] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dbb500c360d4412d8029050e6ab674cc with cas_config_opid_index 39: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261120 15:01:11.011566 9671 catalog_manager.cc:5856] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf reported cstate change: config changed from index 39 to 40, NON_VOTER a962c33db5c04a62863302aa9cd43092 (127.6.176.65) added. New cstate: current_term: 1 leader_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20261120 15:01:11.012265 9943 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Committing config change with OpId 1.40: config changed from index 39 to 40, NON_VOTER a962c33db5c04a62863302aa9cd43092 (127.6.176.65) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } } }
W20261120 15:01:11.013408 10009 consensus_peers.cc:597] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf -> Peer a962c33db5c04a62863302aa9cd43092 (127.6.176.65:44497): Couldn't send request to peer a962c33db5c04a62863302aa9cd43092. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dbb500c360d4412d8029050e6ab674cc. This is attempt 1: this message will repeat every 5th retry.
I20261120 15:01:11.511868 10324 ts_tablet_manager.cc:979] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Initiating tablet copy from peer 8b3a10a9b9ce4deb9d7fda3d9f04a5bf (127.6.176.67:45905)
I20261120 15:01:11.512480 10324 tablet_copy_client.cc:323] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: tablet copy: Beginning tablet copy session from remote peer at address 127.6.176.67:45905
I20261120 15:01:11.516446 10095 tablet_copy_service.cc:140] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Received BeginTabletCopySession request for tablet dbb500c360d4412d8029050e6ab674cc from peer a962c33db5c04a62863302aa9cd43092 ({username='slave'} at 127.6.176.65:59901)
I20261120 15:01:11.516598 10095 tablet_copy_service.cc:161] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Beginning new tablet copy session on tablet dbb500c360d4412d8029050e6ab674cc from peer a962c33db5c04a62863302aa9cd43092 at {username='slave'} at 127.6.176.65:59901: session id = a962c33db5c04a62863302aa9cd43092-dbb500c360d4412d8029050e6ab674cc
I20261120 15:01:11.518126 10095 tablet_copy_source_session.cc:215] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Tablet Copy: opened 5 blocks and 3 log segments
I20261120 15:01:11.522238 10324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb500c360d4412d8029050e6ab674cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:11.527086 10324 tablet_copy_client.cc:806] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: tablet copy: Starting download of 5 data blocks...
I20261120 15:01:11.534215 10324 tablet_copy_client.cc:670] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: tablet copy: Starting download of 3 WAL segments...
I20261120 15:01:11.542218 10324 tablet_copy_client.cc:538] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261120 15:01:11.544517 10324 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Bootstrap starting.
I20261120 15:01:11.551733 10324 log.cc:826] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:11.558497 10324 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=7} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20261120 15:01:11.562187 10324 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 2 replicates
I20261120 15:01:11.564623 10324 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Bootstrap replayed 3/3 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20261120 15:01:11.564963 10324 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Bootstrap complete.
I20261120 15:01:11.565665 10324 ts_tablet_manager.cc:1449] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.006s
I20261120 15:01:11.567682 10324 raft_consensus.cc:366] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } }
I20261120 15:01:11.567900 10324 raft_consensus.cc:787] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a962c33db5c04a62863302aa9cd43092, State: Initialized, Role: LEARNER
I20261120 15:01:11.568246 10324 consensus_queue.cc:261] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 1.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: NON_VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: true } }
I20261120 15:01:11.568997 10324 ts_tablet_manager.cc:1480] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Time spent starting tablet: real 0.003s user 0.002s sys 0.003s
I20261120 15:01:11.569823 10094 tablet_copy_service.cc:342] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Request end of tablet copy session a962c33db5c04a62863302aa9cd43092-dbb500c360d4412d8029050e6ab674cc received from {username='slave'} at 127.6.176.65:59901
I20261120 15:01:11.569968 10094 tablet_copy_service.cc:434] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: ending tablet copy session a962c33db5c04a62863302aa9cd43092-dbb500c360d4412d8029050e6ab674cc on tablet dbb500c360d4412d8029050e6ab674cc with peer a962c33db5c04a62863302aa9cd43092
I20261120 15:01:11.571413 9860 maintenance_manager.cc:419] P a962c33db5c04a62863302aa9cd43092: Scheduling UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc): 549 bytes on disk
I20261120 15:01:11.572242 9750 maintenance_manager.cc:643] P a962c33db5c04a62863302aa9cd43092: UndoDeltaBlockGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20261120 15:01:12.077559 9813 raft_consensus.cc:1279] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.39->[1.40-1.40] Dedup: 1.40->[]
I20261120 15:01:12.155202 9991 maintenance_manager.cc:419] P b3293a34669c4d5ba10faa4bb5340e61: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 1577906 bytes of WAL
I20261120 15:01:12.155535 9881 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 2 log segments from log reader
I20261120 15:01:12.155628 9881 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000004 (ops 21-25)
I20261120 15:01:12.155686 9881 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000005 (ops 26-32)
I20261120 15:01:12.156231 9881 maintenance_manager.cc:643] P b3293a34669c4d5ba10faa4bb5340e61: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:12.157488 10122 maintenance_manager.cc:419] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 1840912 bytes of WAL
I20261120 15:01:12.157727 10012 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 2 log segments from log reader
I20261120 15:01:12.157821 10012 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000004 (ops 23-29)
I20261120 15:01:12.157878 10012 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000005 (ops 30-36)
I20261120 15:01:12.158425 10012 maintenance_manager.cc:643] P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
W20261120 15:01:12.218215 9658 catalog_manager.cc:4808] TS 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dbb500c360d4412d8029050e6ab674cc: Network error: Client connection negotiation failed: client connection to 127.6.176.68:36605: connect: Connection refused (error 111)
I20261120 15:01:12.698597 10332 raft_consensus.cc:1122] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: attempting to promote NON_VOTER a962c33db5c04a62863302aa9cd43092 to VOTER
I20261120 15:01:12.699020 10332 consensus_queue.cc:238] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 1.40, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false } }
I20261120 15:01:12.700361 9813 raft_consensus.cc:1338] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [term 1 LEARNER]: Refusing update from remote peer 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20261120 15:01:12.700397 9943 raft_consensus.cc:1338] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Refusing update from remote peer 8b3a10a9b9ce4deb9d7fda3d9f04a5bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 40. Preceding OpId from leader: term: 1 index: 41. (index mismatch)
I20261120 15:01:12.700850 10338 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261120 15:01:12.701059 10340 consensus_queue.cc:1049] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
I20261120 15:01:12.702857 10340 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf [term 1 LEADER]: Committing config change with OpId 1.41: config changed from index 40 to 41, a962c33db5c04a62863302aa9cd43092 (127.6.176.65) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false } } }
I20261120 15:01:12.703260 9943 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P b3293a34669c4d5ba10faa4bb5340e61 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, a962c33db5c04a62863302aa9cd43092 (127.6.176.65) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false } } }
I20261120 15:01:12.704618 9669 catalog_manager.cc:5856] T dbb500c360d4412d8029050e6ab674cc P 8b3a10a9b9ce4deb9d7fda3d9f04a5bf reported cstate change: config changed from index 40 to 41, a962c33db5c04a62863302aa9cd43092 (127.6.176.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20261120 15:01:12.704659 9813 raft_consensus.cc:3183] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092 [term 1 FOLLOWER]: Committing config change with OpId 1.41: config changed from index 40 to 41, a962c33db5c04a62863302aa9cd43092 (127.6.176.65) changed from NON_VOTER to VOTER. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "a962c33db5c04a62863302aa9cd43092" member_type: VOTER last_known_addr { host: "127.6.176.65" port: 44497 } attrs { promote: false } } }
I20261120 15:01:12.760251 6849 raft_consensus-itest-base.cc:295] Servers converged with original term 1
I20261120 15:01:12.760316 6849 raft_consensus-itest-base.cc:303] Restarting 4bf3c4e5628248e085e90354d74818e2
I20261120 15:01:12.760655 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:36605
--local_ip_for_outbound_sockets=127.6.176.68
--tserver_master_addrs=127.6.176.126:38127
--webserver_port=38313
--webserver_interface=127.6.176.68
--builtin_ntp_servers=127.6.176.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:12.776929 9860 maintenance_manager.cc:419] P a962c33db5c04a62863302aa9cd43092: Scheduling LogGCOp(dbb500c360d4412d8029050e6ab674cc): free 1840860 bytes of WAL
I20261120 15:01:12.777359 9750 log_reader.cc:385] T dbb500c360d4412d8029050e6ab674cc: removed 2 log segments from log reader
I20261120 15:01:12.777436 9750 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000001 (ops 23-29)
I20261120 15:01:12.777483 9750 log.cc:1079] T dbb500c360d4412d8029050e6ab674cc P a962c33db5c04a62863302aa9cd43092: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/wals/dbb500c360d4412d8029050e6ab674cc/wal-000000002 (ops 30-36)
I20261120 15:01:12.778167 9750 maintenance_manager.cc:643] P a962c33db5c04a62863302aa9cd43092: LogGCOp(dbb500c360d4412d8029050e6ab674cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
W20261120 15:01:12.865767 10351 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:12.865971 10351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:12.865993 10351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:12.866035 10351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:12.869410 10351 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:12.869472 10351 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:12.869508 10351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:12.869572 10351 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:12.869602 10351 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:01:12.873430 10351 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:36605
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=38313
--tserver_master_addrs=127.6.176.126:38127
--never_fsync=true
--heap_profile_path=/tmp/kudu.10351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:12.874256 10351 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:12.874966 10351 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:12.880582 10357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:12.880607 10359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:12.880611 10356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:12.880760 10351 server_base.cc:1061] running on GCE node
I20261120 15:01:12.881176 10351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:12.881718 10351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:12.882897 10351 hybrid_clock.cc:648] HybridClock initialized: now 1795186872882882 us; error 30 us; skew 500 ppm
I20261120 15:01:12.884711 10351 webserver.cc:577] Webserver started at http://127.6.176.68:38313/ using document root <none> and password file <none>
I20261120 15:01:12.885306 10351 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:12.885399 10351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:12.888792 10351 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:12.890882 10365 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:12.891928 10351 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:12.892058 10351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4bf3c4e5628248e085e90354d74818e2"
format_stamp: "Formatted at 2026-11-20 15:01:10 on dist-test-slave-m239"
I20261120 15:01:12.892452 10351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:12.912079 10351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:12.912560 10351 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:12.912736 10351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:12.913275 10351 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:12.914471 10372 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261120 15:01:12.918015 10351 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 15:01:12.918092 10351 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20261120 15:01:12.918221 10351 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 15:01:12.919826 10351 ts_tablet_manager.cc:662] Registered 1 tablets
I20261120 15:01:12.919891 10351 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20261120 15:01:12.920222 10372 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Bootstrap starting.
I20261120 15:01:12.928632 10372 log.cc:826] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:12.930570 10351 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:36605
I20261120 15:01:12.930604 10480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:36605 every 8 connection(s)
I20261120 15:01:12.931099 10372 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 15:01:12.931535 10372 tablet_bootstrap.cc:492] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Bootstrap complete.
I20261120 15:01:12.931721 10351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261120 15:01:12.932008 10372 ts_tablet_manager.cc:1449] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20261120 15:01:12.935500 10372 raft_consensus.cc:366] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:12.935832 10372 raft_consensus.cc:787] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf3c4e5628248e085e90354d74818e2, State: Initialized, Role: FOLLOWER
I20261120 15:01:12.936229 10372 consensus_queue.cc:261] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b3a10a9b9ce4deb9d7fda3d9f04a5bf" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 45905 } } peers { permanent_uuid: "b3293a34669c4d5ba10faa4bb5340e61" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 40691 } } peers { permanent_uuid: "4bf3c4e5628248e085e90354d74818e2" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 36605 } }
I20261120 15:01:12.936800 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10351
I20261120 15:01:12.937036 10372 ts_tablet_manager.cc:1480] T dbb500c360d4412d8029050e6ab674cc P 4bf3c4e5628248e085e90354d74818e2: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20261120 15:01:12.939937 10481 heartbeater.cc:345] Connected to a master server at 127.6.176.126:38127
I20261120 15:01:12.940168 10481 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:12.940722 10481 heartbeater.cc:508] Master 127.6.176.126:38127 requested a full tablet report, sending...
I20261120 15:01:12.941506 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9731
I20261120 15:01:12.942736 9671 ts_manager.cc:210] Re-registered known tserver with Master: 4bf3c4e5628248e085e90354d74818e2 (127.6.176.68:36605)
I20261120 15:01:12.943722 9671 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:53061
I20261120 15:01:12.945350 10481 heartbeater.cc:500] Master 127.6.176.126:38127 was elected leader, sending a full tablet report...
I20261120 15:01:12.948644 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9862
I20261120 15:01:12.956295 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9993
I20261120 15:01:12.963436 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10351
I20261120 15:01:12.969004 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 9640
2026-11-20T15:01:12Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestFallBehind (3707 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
I20261120 15:01:12.993551 6849 ts_itest-base.cc:115] Starting cluster with:
I20261120 15:01:12.993592 6849 ts_itest-base.cc:116] --------------
I20261120 15:01:12.993604 6849 ts_itest-base.cc:117] 4 tablet servers
I20261120 15:01:12.993615 6849 ts_itest-base.cc:118] 3 replicas per TS
I20261120 15:01:12.993651 6849 ts_itest-base.cc:119] --------------
2026-11-20T15:01:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-20T15:01:12Z Disabled control of system clock
I20261120 15:01:13.001564 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:43057
--webserver_interface=127.6.176.126
--webserver_port=0
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.176.126:43057 with env {}
W20261120 15:01:13.102541 10493 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:13.102746 10493 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:13.102857 10493 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:13.106125 10493 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261120 15:01:13.106215 10493 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:13.106259 10493 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261120 15:01:13.106328 10493 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261120 15:01:13.110143 10493 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.176.126:43057
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.176.126:43057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.176.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:13.110916 10493 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:13.111640 10493 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:13.116492 10498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.116597 10499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.116705 10493 server_base.cc:1061] running on GCE node
W20261120 15:01:13.116539 10501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.117267 10493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:13.118114 10493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:13.119277 10493 hybrid_clock.cc:648] HybridClock initialized: now 1795186873119257 us; error 40 us; skew 500 ppm
I20261120 15:01:13.120864 10493 webserver.cc:577] Webserver started at http://127.6.176.126:41529/ using document root <none> and password file <none>
I20261120 15:01:13.121429 10493 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:13.121479 10493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:13.121714 10493 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:13.122975 10493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6887278305d6469ab5b3e58f77bf51a5"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.123503 10493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6887278305d6469ab5b3e58f77bf51a5"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.126709 10493 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20261120 15:01:13.128916 10507 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.129869 10493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:01:13.130033 10493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6887278305d6469ab5b3e58f77bf51a5"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.130134 10493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:13.170333 10493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:13.170902 10493 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:13.171092 10493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:13.178164 10493 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.126:43057
I20261120 15:01:13.178210 10559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.126:43057 every 8 connection(s)
I20261120 15:01:13.179219 10493 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261120 15:01:13.182369 10560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:13.186846 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10493
I20261120 15:01:13.186990 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261120 15:01:13.187547 10560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: Bootstrap starting.
I20261120 15:01:13.190306 10560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:13.191066 10560 log.cc:826] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:13.192447 10560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: No bootstrap required, opened a new log
I20261120 15:01:13.195228 10560 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } }
I20261120 15:01:13.195437 10560 raft_consensus.cc:392] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:13.195474 10560 raft_consensus.cc:787] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6887278305d6469ab5b3e58f77bf51a5, State: Initialized, Role: FOLLOWER
I20261120 15:01:13.195811 10560 consensus_queue.cc:261] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } }
I20261120 15:01:13.195920 10560 raft_consensus.cc:407] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261120 15:01:13.195962 10560 raft_consensus.cc:534] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261120 15:01:13.196022 10560 raft_consensus.cc:3296] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:13.196270 10560 raft_consensus.cc:556] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } }
I20261120 15:01:13.197171 10560 leader_election.cc:304] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6887278305d6469ab5b3e58f77bf51a5; no voters:
I20261120 15:01:13.197377 10560 leader_election.cc:290] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261120 15:01:13.197453 10565 raft_consensus.cc:3024] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:13.197688 10565 raft_consensus.cc:744] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6887278305d6469ab5b3e58f77bf51a5, State: Running, Role: LEADER
I20261120 15:01:13.198017 10565 consensus_queue.cc:238] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } }
I20261120 15:01:13.198447 10560 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [sys.catalog]: configured and running, proceeding with master startup.
I20261120 15:01:13.199674 10567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6887278305d6469ab5b3e58f77bf51a5. Latest consensus state: current_term: 1 leader_uuid: "6887278305d6469ab5b3e58f77bf51a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } } }
I20261120 15:01:13.199776 10566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6887278305d6469ab5b3e58f77bf51a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6887278305d6469ab5b3e58f77bf51a5" member_type: VOTER last_known_addr { host: "127.6.176.126" port: 43057 } } }
I20261120 15:01:13.199785 10567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [sys.catalog]: This master's current role is: LEADER
I20261120 15:01:13.199879 10566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5 [sys.catalog]: This master's current role is: LEADER
W20261120 15:01:13.203008 10581 catalog_manager.cc:1609] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20261120 15:01:13.203068 10581 catalog_manager.cc:898] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20261120 15:01:13.203151 10574 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261120 15:01:13.203953 10574 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261120 15:01:13.208886 10574 catalog_manager.cc:1398] Generated new cluster ID: d99068d26b4845e6892ff63ea25b6d8d
I20261120 15:01:13.208948 10574 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261120 15:01:13.220937 10574 catalog_manager.cc:1421] Generated new certificate authority record
I20261120 15:01:13.221853 10574 catalog_manager.cc:1555] Loading token signing keys...
I20261120 15:01:13.233438 10574 catalog_manager.cc:6237] T 00000000000000000000000000000000 P 6887278305d6469ab5b3e58f77bf51a5: Generated new TSK 0
I20261120 15:01:13.234125 10574 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261120 15:01:13.241456 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.65:0
--local_ip_for_outbound_sockets=127.6.176.65
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261120 15:01:13.353785 10584 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:13.354065 10584 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:13.354141 10584 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:13.354262 10584 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:13.357765 10584 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:13.357848 10584 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:13.357924 10584 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:13.358057 10584 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:13.358129 10584 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.65
I20261120 15:01:13.362082 10584 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.176.65
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--never_fsync=true
--heap_profile_path=/tmp/kudu.10584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:13.362903 10584 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:13.363644 10584 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:13.369320 10590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.369331 10592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.369494 10584 server_base.cc:1061] running on GCE node
W20261120 15:01:13.369320 10589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.369993 10584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:13.370548 10584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:13.371716 10584 hybrid_clock.cc:648] HybridClock initialized: now 1795186873371691 us; error 38 us; skew 500 ppm
I20261120 15:01:13.377060 10584 webserver.cc:577] Webserver started at http://127.6.176.65:34659/ using document root <none> and password file <none>
I20261120 15:01:13.378079 10584 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:13.378188 10584 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:13.378490 10584 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:13.380282 10584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e26474d419084a978855e2d9afd89f6d"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.380869 10584 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e26474d419084a978855e2d9afd89f6d"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.384914 10584 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:13.387285 10598 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.388365 10584 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:13.388494 10584 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e26474d419084a978855e2d9afd89f6d"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.388593 10584 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:13.404580 10584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:13.405036 10584 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:13.405211 10584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:13.405681 10584 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:13.406971 10584 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:13.407032 10584 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.407090 10584 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:13.407124 10584 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.415730 10584 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.65:40111
I20261120 15:01:13.415814 10711 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.65:40111 every 8 connection(s)
I20261120 15:01:13.416639 10584 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261120 15:01:13.417658 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10584
I20261120 15:01:13.417788 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261120 15:01:13.420217 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.66:0
--local_ip_for_outbound_sockets=127.6.176.66
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:13.425375 10712 heartbeater.cc:345] Connected to a master server at 127.6.176.126:43057
I20261120 15:01:13.425649 10712 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:13.426318 10712 heartbeater.cc:508] Master 127.6.176.126:43057 requested a full tablet report, sending...
I20261120 15:01:13.428143 10524 ts_manager.cc:210] Registered new tserver with Master: e26474d419084a978855e2d9afd89f6d (127.6.176.65:40111)
I20261120 15:01:13.429407 10524 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.65:44945
W20261120 15:01:13.523957 10715 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:13.524139 10715 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:13.524256 10715 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:13.524320 10715 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:13.527798 10715 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:13.527848 10715 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:13.527882 10715 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:13.528017 10715 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:13.528052 10715 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.66
I20261120 15:01:13.531931 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.176.66
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--never_fsync=true
--heap_profile_path=/tmp/kudu.10715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:13.532735 10715 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:13.533476 10715 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:13.539095 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.539146 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.539147 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.539376 10715 server_base.cc:1061] running on GCE node
I20261120 15:01:13.539709 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:13.540257 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:13.541422 10715 hybrid_clock.cc:648] HybridClock initialized: now 1795186873541400 us; error 39 us; skew 500 ppm
I20261120 15:01:13.543124 10715 webserver.cc:577] Webserver started at http://127.6.176.66:37645/ using document root <none> and password file <none>
I20261120 15:01:13.543735 10715 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:13.543808 10715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:13.544042 10715 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:13.545306 10715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.545774 10715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.548918 10715 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:13.551133 10729 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.552147 10715 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:13.552284 10715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.552384 10715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:13.564541 10715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:13.565009 10715 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:13.565178 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:13.565681 10715 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:13.566591 10715 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:13.566668 10715 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.566744 10715 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:13.566802 10715 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.575299 10715 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.66:45435
I20261120 15:01:13.575325 10842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.66:45435 every 8 connection(s)
I20261120 15:01:13.576267 10715 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261120 15:01:13.583880 10843 heartbeater.cc:345] Connected to a master server at 127.6.176.126:43057
I20261120 15:01:13.584108 10843 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:13.584560 10843 heartbeater.cc:508] Master 127.6.176.126:43057 requested a full tablet report, sending...
I20261120 15:01:13.585570 10524 ts_manager.cc:210] Registered new tserver with Master: ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66:45435)
I20261120 15:01:13.586127 10524 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.66:45717
I20261120 15:01:13.586342 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10715
I20261120 15:01:13.586479 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261120 15:01:13.588507 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:0
--local_ip_for_outbound_sockets=127.6.176.67
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261120 15:01:13.688620 10846 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:13.688894 10846 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:13.688961 10846 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:13.689052 10846 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:13.692493 10846 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:13.692574 10846 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:13.692652 10846 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:13.692803 10846 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:13.692876 10846 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:13.696928 10846 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--never_fsync=true
--heap_profile_path=/tmp/kudu.10846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:13.697769 10846 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:13.698477 10846 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:13.704123 10852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.704206 10851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.704312 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.704486 10846 server_base.cc:1061] running on GCE node
I20261120 15:01:13.704824 10846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:13.705354 10846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:13.706499 10846 hybrid_clock.cc:648] HybridClock initialized: now 1795186873706480 us; error 38 us; skew 500 ppm
I20261120 15:01:13.708194 10846 webserver.cc:577] Webserver started at http://127.6.176.67:45469/ using document root <none> and password file <none>
I20261120 15:01:13.708760 10846 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:13.708808 10846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:13.709050 10846 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:13.710312 10846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.710755 10846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.713955 10846 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20261120 15:01:13.716257 10860 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.717212 10846 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:13.717341 10846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.717439 10846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:13.739614 10846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:13.740085 10846 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:13.740275 10846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:13.740751 10846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:13.741675 10846 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:13.741743 10846 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.741818 10846 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:13.741871 10846 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.750352 10846 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:44081
I20261120 15:01:13.750380 10973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:44081 every 8 connection(s)
I20261120 15:01:13.751271 10846 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261120 15:01:13.753822 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10846
I20261120 15:01:13.753935 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261120 15:01:13.756270 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.68:0
--local_ip_for_outbound_sockets=127.6.176.68
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261120 15:01:13.758997 10974 heartbeater.cc:345] Connected to a master server at 127.6.176.126:43057
I20261120 15:01:13.759225 10974 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:13.759829 10974 heartbeater.cc:508] Master 127.6.176.126:43057 requested a full tablet report, sending...
I20261120 15:01:13.760982 10524 ts_manager.cc:210] Registered new tserver with Master: 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081)
I20261120 15:01:13.761520 10524 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:44379
W20261120 15:01:13.861837 10977 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:13.862035 10977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:13.862226 10977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:13.862313 10977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:13.865831 10977 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:13.865883 10977 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:13.865918 10977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:13.866046 10977 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:13.866080 10977 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.68
I20261120 15:01:13.869904 10977 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.176.68
--webserver_port=0
--tserver_master_addrs=127.6.176.126:43057
--never_fsync=true
--heap_profile_path=/tmp/kudu.10977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.68
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:13.870645 10977 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:13.871429 10977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:13.876799 10983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.876856 10985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:13.876803 10982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:13.876958 10977 server_base.cc:1061] running on GCE node
I20261120 15:01:13.877417 10977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:13.877904 10977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:13.879070 10977 hybrid_clock.cc:648] HybridClock initialized: now 1795186873879050 us; error 37 us; skew 500 ppm
I20261120 15:01:13.880722 10977 webserver.cc:577] Webserver started at http://127.6.176.68:46825/ using document root <none> and password file <none>
I20261120 15:01:13.881273 10977 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:13.881346 10977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:13.881556 10977 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261120 15:01:13.882772 10977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ce2764a03952415f8a7bf9f1e33b6254"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.883224 10977 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ce2764a03952415f8a7bf9f1e33b6254"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.886296 10977 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20261120 15:01:13.888589 10991 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.889500 10977 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20261120 15:01:13.889632 10977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ce2764a03952415f8a7bf9f1e33b6254"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:13.889729 10977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:13.913156 10977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:13.913651 10977 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:13.913810 10977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:13.914322 10977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:13.915241 10977 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261120 15:01:13.915310 10977 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.915396 10977 ts_tablet_manager.cc:662] Registered 0 tablets
I20261120 15:01:13.915441 10977 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:13.924129 10977 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.68:37919
I20261120 15:01:13.924158 11104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.68:37919 every 8 connection(s)
I20261120 15:01:13.925057 10977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261120 15:01:13.932052 11105 heartbeater.cc:345] Connected to a master server at 127.6.176.126:43057
I20261120 15:01:13.932287 11105 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:13.932343 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 10977
I20261120 15:01:13.932456 6849 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261120 15:01:13.932811 11105 heartbeater.cc:508] Master 127.6.176.126:43057 requested a full tablet report, sending...
I20261120 15:01:13.933817 10524 ts_manager.cc:210] Registered new tserver with Master: ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:13.934387 10524 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.68:59039
I20261120 15:01:13.934772 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:13.945865 10524 catalog_manager.cc:2301] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44268:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261120 15:01:13.965216 10908 tablet_service.cc:1495] Processing CreateTablet for tablet 2083630ce86b4ffb80009e7727f50af7 (DEFAULT_TABLE table=TestTable [id=309006bac3264fabb731df24176bbbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:13.965240 10777 tablet_service.cc:1495] Processing CreateTablet for tablet 2083630ce86b4ffb80009e7727f50af7 (DEFAULT_TABLE table=TestTable [id=309006bac3264fabb731df24176bbbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:13.965240 11039 tablet_service.cc:1495] Processing CreateTablet for tablet 2083630ce86b4ffb80009e7727f50af7 (DEFAULT_TABLE table=TestTable [id=309006bac3264fabb731df24176bbbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261120 15:01:13.966020 10908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2083630ce86b4ffb80009e7727f50af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:13.966109 11039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2083630ce86b4ffb80009e7727f50af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:13.966109 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2083630ce86b4ffb80009e7727f50af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20261120 15:01:13.971419 11122 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Bootstrap starting.
I20261120 15:01:13.972486 11123 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Bootstrap starting.
I20261120 15:01:13.973973 11122 tablet_bootstrap.cc:654] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:13.974606 11123 tablet_bootstrap.cc:654] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:13.974782 11122 log.cc:826] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:13.975422 11124 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Bootstrap starting.
I20261120 15:01:13.975528 11123 log.cc:826] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:13.976362 11122 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: No bootstrap required, opened a new log
I20261120 15:01:13.976581 11122 ts_tablet_manager.cc:1449] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:13.976835 11123 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: No bootstrap required, opened a new log
I20261120 15:01:13.977036 11123 ts_tablet_manager.cc:1449] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20261120 15:01:13.977695 11124 tablet_bootstrap.cc:654] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Neither blocks nor log segments found. Creating new log.
I20261120 15:01:13.978513 11124 log.cc:826] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:13.979780 11123 raft_consensus.cc:366] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.979916 11124 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: No bootstrap required, opened a new log
I20261120 15:01:13.979970 11123 raft_consensus.cc:392] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:13.979864 11122 raft_consensus.cc:366] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.980026 11123 raft_consensus.cc:787] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2764a03952415f8a7bf9f1e33b6254, State: Initialized, Role: FOLLOWER
I20261120 15:01:13.980042 11122 raft_consensus.cc:392] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:13.980080 11124 ts_tablet_manager.cc:1449] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261120 15:01:13.980091 11122 raft_consensus.cc:787] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab52e10dd3d4b95b4fc6922332cf485, State: Initialized, Role: FOLLOWER
I20261120 15:01:13.980484 11123 consensus_queue.cc:261] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.980506 11122 consensus_queue.cc:261] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.981169 11123 ts_tablet_manager.cc:1480] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20261120 15:01:13.981256 11122 ts_tablet_manager.cc:1480] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20261120 15:01:13.981564 11105 heartbeater.cc:500] Master 127.6.176.126:43057 was elected leader, sending a full tablet report...
I20261120 15:01:13.982818 11124 raft_consensus.cc:366] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.983004 11124 raft_consensus.cc:392] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261120 15:01:13.983052 11124 raft_consensus.cc:787] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba72d5ec0c0d47b2b17396e5d22cc3de, State: Initialized, Role: FOLLOWER
I20261120 15:01:13.983434 11124 consensus_queue.cc:261] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.983491 10974 heartbeater.cc:500] Master 127.6.176.126:43057 was elected leader, sending a full tablet report...
I20261120 15:01:13.983537 11129 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:01:13.983642 11129 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.984187 11124 ts_tablet_manager.cc:1480] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20261120 15:01:13.984516 10843 heartbeater.cc:500] Master 127.6.176.126:43057 was elected leader, sending a full tablet report...
I20261120 15:01:13.985963 11129 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66:45435), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:13.989871 10797 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" is_pre_election: true
I20261120 15:01:13.990116 10797 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 3ab52e10dd3d4b95b4fc6922332cf485 in term 0
I20261120 15:01:13.990302 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254" is_pre_election: true
I20261120 15:01:13.990523 10863 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab52e10dd3d4b95b4fc6922332cf485, ba72d5ec0c0d47b2b17396e5d22cc3de; no voters:
I20261120 15:01:13.990557 11059 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 0 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate 3ab52e10dd3d4b95b4fc6922332cf485 in term 0
I20261120 15:01:13.990743 11129 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261120 15:01:13.990818 11129 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:01:13.990872 11129 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:13.991209 11129 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.992347 11129 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [CANDIDATE]: Term 1 election: Requested vote from peers ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66:45435), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:13.992630 10797 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de"
I20261120 15:01:13.992717 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254"
I20261120 15:01:13.992754 10797 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:13.992857 11059 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 0 FOLLOWER]: Advancing to term 1
I20261120 15:01:13.993112 11059 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 3ab52e10dd3d4b95b4fc6922332cf485 in term 1
I20261120 15:01:13.993479 10797 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Leader election vote request: granting yes vote for candidate 3ab52e10dd3d4b95b4fc6922332cf485 in term 1
I20261120 15:01:13.994514 10863 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ab52e10dd3d4b95b4fc6922332cf485, ba72d5ec0c0d47b2b17396e5d22cc3de; no voters:
I20261120 15:01:13.994707 11129 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Leader election won for term 1
I20261120 15:01:13.994841 11129 raft_consensus.cc:744] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 LEADER]: Becoming Leader. State: Replica: 3ab52e10dd3d4b95b4fc6922332cf485, State: Running, Role: LEADER
I20261120 15:01:13.995184 11129 consensus_queue.cc:238] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:13.998495 11129 consensus_queue.cc:1049] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:13.998687 11132 consensus_queue.cc:1049] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261120 15:01:13.998555 10522 catalog_manager.cc:5856] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67). New cstate: current_term: 1 leader_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } health_report { overall_health: UNKNOWN } } }
I20261120 15:01:14.009666 6849 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261120 15:01:14.011274 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ba72d5ec0c0d47b2b17396e5d22cc3de to finish bootstrapping
I20261120 15:01:14.017066 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3ab52e10dd3d4b95b4fc6922332cf485 to finish bootstrapping
I20261120 15:01:14.021734 6849 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce2764a03952415f8a7bf9f1e33b6254 to finish bootstrapping
I20261120 15:01:14.027839 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10846
I20261120 15:01:14.034543 6849 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
/tmp/dist-test-taskkAkmHg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.176.67:44081
--local_ip_for_outbound_sockets=127.6.176.67
--tserver_master_addrs=127.6.176.126:43057
--webserver_port=45469
--webserver_interface=127.6.176.67
--builtin_ntp_servers=127.6.176.84:36545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=3000 with env {}
W20261120 15:01:14.136466 11141 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261120 15:01:14.136660 11141 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261120 15:01:14.136706 11141 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261120 15:01:14.136792 11141 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=3000
W20261120 15:01:14.136824 11141 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261120 15:01:14.140537 11141 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261120 15:01:14.140589 11141 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261120 15:01:14.140623 11141 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261120 15:01:14.140746 11141 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261120 15:01:14.140780 11141 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.176.67
I20261120 15:01:14.144771 11141 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.176.84:36545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.176.67:44081
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.176.67
--webserver_port=45469
--tserver_master_addrs=127.6.176.126:43057
--tablet_bootstrap_inject_latency_ms=3000
--never_fsync=true
--heap_profile_path=/tmp/kudu.11141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.176.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 05464f8854a90597ddf8dfd45c2061c63b0b716b
build type DEBUG
built by None at 20 Nov 2026 14:43:14 UTC on bdcb31816ec0
build id 14875
I20261120 15:01:14.145519 11141 env_posix.cc:2337] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261120 15:01:14.146221 11141 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261120 15:01:14.151840 11146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261120 15:01:14.151782 11149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:14.151896 11141 server_base.cc:1061] running on GCE node
W20261120 15:01:14.151782 11147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261120 15:01:14.152308 11141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261120 15:01:14.152817 11141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261120 15:01:14.153960 11141 hybrid_clock.cc:648] HybridClock initialized: now 1795186874153933 us; error 39 us; skew 500 ppm
I20261120 15:01:14.155718 11141 webserver.cc:577] Webserver started at http://127.6.176.67:45469/ using document root <none> and password file <none>
I20261120 15:01:14.156332 11141 fs_manager.cc:362] Metadata directory not provided
I20261120 15:01:14.156394 11141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261120 15:01:14.159541 11141 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20261120 15:01:14.161540 11155 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261120 15:01:14.162474 11141 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:14.162564 11141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
format_stamp: "Formatted at 2026-11-20 15:01:13 on dist-test-slave-m239"
I20261120 15:01:14.162933 11141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261120 15:01:14.176222 11141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261120 15:01:14.176617 11141 env_posix.cc:2337] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261120 15:01:14.176767 11141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261120 15:01:14.177232 11141 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261120 15:01:14.178417 11162 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261120 15:01:14.181842 11141 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261120 15:01:14.181900 11141 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20261120 15:01:14.181967 11141 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261120 15:01:14.183465 11141 ts_tablet_manager.cc:662] Registered 1 tablets
W20261120 15:01:14.183493 11162 ts_tablet_manager.cc:1428] Injecting 3000ms delay in tablet bootstrapping
I20261120 15:01:14.183518 11141 ts_tablet_manager.cc:641] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20261120 15:01:14.193902 11141 rpc_server.cc:307] RPC server started. Bound to: 127.6.176.67:44081
I20261120 15:01:14.193928 11269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.176.67:44081 every 8 connection(s)
I20261120 15:01:14.194919 11141 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261120 15:01:14.200151 6849 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu as pid 11141
I20261120 15:01:14.202759 11270 heartbeater.cc:345] Connected to a master server at 127.6.176.126:43057
I20261120 15:01:14.202993 11270 heartbeater.cc:462] Registering TS with master...
I20261120 15:01:14.203531 11270 heartbeater.cc:508] Master 127.6.176.126:43057 requested a full tablet report, sending...
I20261120 15:01:14.204926 10524 ts_manager.cc:210] Re-registered known tserver with Master: 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081)
I20261120 15:01:14.205812 10524 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.6.176.67:52937
I20261120 15:01:14.432018 10712 heartbeater.cc:500] Master 127.6.176.126:43057 was elected leader, sending a full tablet report...
I20261120 15:01:15.208019 11270 heartbeater.cc:500] Master 127.6.176.126:43057 was elected leader, sending a full tablet report...
I20261120 15:01:15.504189 11275 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ab52e10dd3d4b95b4fc6922332cf485)
I20261120 15:01:15.504235 11276 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3ab52e10dd3d4b95b4fc6922332cf485)
I20261120 15:01:15.504344 11275 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:15.504364 11276 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:15.505489 11275 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:15.505849 11276 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66:45435), 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081)
I20261120 15:01:15.510076 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254" is_pre_election: true
I20261120 15:01:15.510193 10797 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ce2764a03952415f8a7bf9f1e33b6254" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" is_pre_election: true
I20261120 15:01:15.510253 11059 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in term 1
I20261120 15:01:15.510349 10797 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate ce2764a03952415f8a7bf9f1e33b6254 in term 1
I20261120 15:01:15.510629 10731 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba72d5ec0c0d47b2b17396e5d22cc3de, ce2764a03952415f8a7bf9f1e33b6254; no voters:
I20261120 15:01:15.510727 10994 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba72d5ec0c0d47b2b17396e5d22cc3de, ce2764a03952415f8a7bf9f1e33b6254; no voters:
I20261120 15:01:15.510929 11276 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 15:01:15.510942 11275 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261120 15:01:15.510999 11275 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3ab52e10dd3d4b95b4fc6922332cf485)
I20261120 15:01:15.511003 11276 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3ab52e10dd3d4b95b4fc6922332cf485)
I20261120 15:01:15.511037 11275 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:01:15.511053 11276 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 1 FOLLOWER]: Advancing to term 2
I20261120 15:01:15.511260 11275 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:15.511307 11276 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:15.511134 11224 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" is_pre_election: true
I20261120 15:01:15.511145 11223 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ce2764a03952415f8a7bf9f1e33b6254" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" is_pre_election: true
I20261120 15:01:15.512557 11276 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [CANDIDATE]: Term 2 election: Requested vote from peers ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66:45435), 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081)
I20261120 15:01:15.512670 11275 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 election: Requested vote from peers 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:15.512960 10797 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ce2764a03952415f8a7bf9f1e33b6254" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de"
I20261120 15:01:15.513121 10797 raft_consensus.cc:2628] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce2764a03952415f8a7bf9f1e33b6254 in current term 2: Already voted for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in this term.
I20261120 15:01:15.513173 11222 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ce2764a03952415f8a7bf9f1e33b6254" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
I20261120 15:01:15.513306 11223 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
W20261120 15:01:15.513298 10733 leader_election.cc:343] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:15.513593 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254"
W20261120 15:01:15.513622 10733 leader_election.cc:343] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081): Illegal state: must be running to vote when last-logged opid is not known
W20261120 15:01:15.513712 10995 leader_election.cc:343] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081): Illegal state: must be running to vote when last-logged opid is not known
I20261120 15:01:15.513794 11059 raft_consensus.cc:2628] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ba72d5ec0c0d47b2b17396e5d22cc3de in current term 2: Already voted for candidate ce2764a03952415f8a7bf9f1e33b6254 in this term.
I20261120 15:01:15.513859 10995 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce2764a03952415f8a7bf9f1e33b6254; no voters: 3ab52e10dd3d4b95b4fc6922332cf485, ba72d5ec0c0d47b2b17396e5d22cc3de
I20261120 15:01:15.514001 11276 raft_consensus.cc:2966] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261120 15:01:15.514146 10731 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba72d5ec0c0d47b2b17396e5d22cc3de; no voters: 3ab52e10dd3d4b95b4fc6922332cf485, ce2764a03952415f8a7bf9f1e33b6254
I20261120 15:01:15.514366 11285 raft_consensus.cc:2966] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261120 15:01:17.183938 11162 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Bootstrap starting.
I20261120 15:01:17.190694 11162 log.cc:826] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Log is configured to *not* fsync() on all Append() calls
I20261120 15:01:17.192945 11162 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261120 15:01:17.193284 11162 tablet_bootstrap.cc:492] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Bootstrap complete.
I20261120 15:01:17.193734 11162 ts_tablet_manager.cc:1449] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20261120 15:01:17.195576 11162 raft_consensus.cc:366] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:17.195847 11162 raft_consensus.cc:787] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab52e10dd3d4b95b4fc6922332cf485, State: Initialized, Role: FOLLOWER
I20261120 15:01:17.196316 11162 consensus_queue.cc:261] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:17.196920 11162 ts_tablet_manager.cc:1480] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20261120 15:01:17.459529 11289 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261120 15:01:17.459656 11289 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:17.460054 11289 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:17.460675 11223 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" is_pre_election: true
I20261120 15:01:17.460675 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254" is_pre_election: true
I20261120 15:01:17.460870 11059 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 2 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in term 2
I20261120 15:01:17.460971 11223 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Leader pre-election vote request: granting yes vote for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in term 1
I20261120 15:01:17.461227 10731 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba72d5ec0c0d47b2b17396e5d22cc3de, ce2764a03952415f8a7bf9f1e33b6254; no voters:
I20261120 15:01:17.461426 11289 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261120 15:01:17.461513 11289 raft_consensus.cc:534] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261120 15:01:17.461581 11289 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 2 FOLLOWER]: Advancing to term 3
I20261120 15:01:17.461826 11289 raft_consensus.cc:556] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:17.462996 11289 leader_election.cc:290] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 3 election: Requested vote from peers 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67:44081), ce2764a03952415f8a7bf9f1e33b6254 (127.6.176.68:37919)
I20261120 15:01:17.463373 11223 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3ab52e10dd3d4b95b4fc6922332cf485"
I20261120 15:01:17.463428 11059 tablet_service.cc:1901] Received RequestConsensusVote() RPC: tablet_id: "2083630ce86b4ffb80009e7727f50af7" candidate_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ce2764a03952415f8a7bf9f1e33b6254"
I20261120 15:01:17.463550 11059 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 2 FOLLOWER]: Advancing to term 3
I20261120 15:01:17.463541 11223 raft_consensus.cc:3296] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 1 FOLLOWER]: Advancing to term 3
I20261120 15:01:17.463860 11059 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in term 3
I20261120 15:01:17.463946 11223 raft_consensus.cc:2062] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 3 FOLLOWER]: Leader election vote request: granting yes vote for candidate ba72d5ec0c0d47b2b17396e5d22cc3de in term 3
I20261120 15:01:17.464895 10731 leader_election.cc:304] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba72d5ec0c0d47b2b17396e5d22cc3de, ce2764a03952415f8a7bf9f1e33b6254; no voters:
I20261120 15:01:17.465073 11289 raft_consensus.cc:3024] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 3 FOLLOWER]: Leader election won for term 3
I20261120 15:01:17.465173 11289 raft_consensus.cc:744] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [term 3 LEADER]: Becoming Leader. State: Replica: ba72d5ec0c0d47b2b17396e5d22cc3de, State: Running, Role: LEADER
I20261120 15:01:17.465626 11289 consensus_queue.cc:238] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } }
I20261120 15:01:17.467880 11059 raft_consensus.cc:1338] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254 [term 3 FOLLOWER]: Refusing update from remote peer ba72d5ec0c0d47b2b17396e5d22cc3de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261120 15:01:17.468533 11223 raft_consensus.cc:1338] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485 [term 3 FOLLOWER]: Refusing update from remote peer ba72d5ec0c0d47b2b17396e5d22cc3de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261120 15:01:17.468721 10522 catalog_manager.cc:5856] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de reported cstate change: term changed from 1 to 3, leader changed from 3ab52e10dd3d4b95b4fc6922332cf485 (127.6.176.67) to ba72d5ec0c0d47b2b17396e5d22cc3de (127.6.176.66). New cstate: current_term: 3 leader_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba72d5ec0c0d47b2b17396e5d22cc3de" member_type: VOTER last_known_addr { host: "127.6.176.66" port: 45435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 } health_report { overall_health: UNKNOWN } } }
I20261120 15:01:17.469122 11289 consensus_queue.cc:1049] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab52e10dd3d4b95b4fc6922332cf485" member_type: VOTER last_known_addr { host: "127.6.176.67" port: 44081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 15:01:17.469273 11290 consensus_queue.cc:1049] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2764a03952415f8a7bf9f1e33b6254" member_type: VOTER last_known_addr { host: "127.6.176.68" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261120 15:01:17.863924 6849 test_workload.cc:406] TestWorkload: Skipping table creation because table TestTable already exists
I20261120 15:01:17.864356 6849 raft_config_change-itest.cc:610] Waiting until we've written at least 4MB...
I20261120 15:01:17.939766 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling FlushMRSOp(2083630ce86b4ffb80009e7727f50af7): 1577960 bytes log retention, and flush 2998537 bytes memory
I20261120 15:01:17.990273 6849 raft_config_change-itest.cc:616] Waiting for log GC on ba72d5ec0c0d47b2b17396e5d22cc3de
I20261120 15:01:17.990996 10844 maintenance_manager.cc:419] P ba72d5ec0c0d47b2b17396e5d22cc3de: Scheduling FlushMRSOp(2083630ce86b4ffb80009e7727f50af7): 4602312 bytes log retention, and flush 6119689 bytes memory
I20261120 15:01:18.004549 11271 maintenance_manager.cc:419] P 3ab52e10dd3d4b95b4fc6922332cf485: Scheduling FlushMRSOp(2083630ce86b4ffb80009e7727f50af7): 3944793 bytes log retention, and flush 6119689 bytes memory
I20261120 15:01:18.034247 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: FlushMRSOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.093s user 0.036s sys 0.004s Metrics: {"bytes_written":135869,"cfile_init":1,"compiler_manager_pool.queue_time_us":229,"dirs.queue_time_us":9270,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":895,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":19,"thread_start_us":231,"threads_started":2}
I20261120 15:01:18.035236 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling LogGCOp(2083630ce86b4ffb80009e7727f50af7): free 2498406 bytes of WAL
I20261120 15:01:18.035588 10996 log_reader.cc:385] T 2083630ce86b4ffb80009e7727f50af7: removed 3 log segments from log reader
I20261120 15:01:18.035693 10996 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000001 (ops 1-9)
I20261120 15:01:18.035817 10996 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000002 (ops 10-14)
I20261120 15:01:18.035908 10996 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000003 (ops 15-21)
I20261120 15:01:18.036818 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: LogGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:18.037139 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling FlushMRSOp(2083630ce86b4ffb80009e7727f50af7): 1840708 bytes log retention, and flush 2998537 bytes memory
I20261120 15:01:18.095363 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: FlushMRSOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.058s user 0.025s sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1272,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":578,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":18}
I20261120 15:01:18.096387 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling LogGCOp(2083630ce86b4ffb80009e7727f50af7): free 920186 bytes of WAL
I20261120 15:01:18.096626 10996 log_reader.cc:385] T 2083630ce86b4ffb80009e7727f50af7: removed 1 log segments from log reader
I20261120 15:01:18.096707 10996 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000004 (ops 22-28)
I20261120 15:01:18.097118 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: LogGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:18.097404 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7): 685 bytes on disk
I20261120 15:01:18.098093 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":103,"lbm_reads_lt_1ms":8}
I20261120 15:01:18.098527 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling CompactRowSetsOp(2083630ce86b4ffb80009e7727f50af7): perf score=0.954010
I20261120 15:01:18.106501 10734 maintenance_manager.cc:643] P ba72d5ec0c0d47b2b17396e5d22cc3de: FlushMRSOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.115s user 0.059s sys 0.004s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":6470,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":253,"threads_started":2}
I20261120 15:01:18.107599 10844 maintenance_manager.cc:419] P ba72d5ec0c0d47b2b17396e5d22cc3de: Scheduling LogGCOp(2083630ce86b4ffb80009e7727f50af7): free 4602312 bytes of WAL
I20261120 15:01:18.107965 10734 log_reader.cc:385] T 2083630ce86b4ffb80009e7727f50af7: removed 5 log segments from log reader
I20261120 15:01:18.108059 10734 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000001 (ops 1-9)
I20261120 15:01:18.108191 10734 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000002 (ops 10-16)
I20261120 15:01:18.108265 10734 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000003 (ops 17-23)
I20261120 15:01:18.108335 10734 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000004 (ops 24-30)
I20261120 15:01:18.108403 10734 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ba72d5ec0c0d47b2b17396e5d22cc3de: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-1/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000005 (ops 31-37)
I20261120 15:01:18.109843 10734 maintenance_manager.cc:643] P ba72d5ec0c0d47b2b17396e5d22cc3de: LogGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:18.110142 10844 maintenance_manager.cc:419] P ba72d5ec0c0d47b2b17396e5d22cc3de: Scheduling UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7): 550 bytes on disk
I20261120 15:01:18.110750 10734 maintenance_manager.cc:643] P ba72d5ec0c0d47b2b17396e5d22cc3de: UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20261120 15:01:18.117045 6849 raft_config_change-itest.cc:620] Log GC complete on ba72d5ec0c0d47b2b17396e5d22cc3de
I20261120 15:01:18.127854 11160 maintenance_manager.cc:643] P 3ab52e10dd3d4b95b4fc6922332cf485: FlushMRSOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.123s user 0.059s sys 0.004s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":9476,"dirs.queue_time_us":16002,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":3917,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":631,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":223,"threads_started":2}
I20261120 15:01:18.128728 11271 maintenance_manager.cc:419] P 3ab52e10dd3d4b95b4fc6922332cf485: Scheduling LogGCOp(2083630ce86b4ffb80009e7727f50af7): free 3944793 bytes of WAL
I20261120 15:01:18.128995 11160 log_reader.cc:385] T 2083630ce86b4ffb80009e7727f50af7: removed 5 log segments from log reader
I20261120 15:01:18.129081 11160 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000001 (ops 1-9)
I20261120 15:01:18.129186 11160 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000002 (ops 10-16)
I20261120 15:01:18.129256 11160 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000003 (ops 17-21)
I20261120 15:01:18.129323 11160 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000004 (ops 22-28)
I20261120 15:01:18.129390 11160 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P 3ab52e10dd3d4b95b4fc6922332cf485: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-2/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000005 (ops 29-32)
I20261120 15:01:18.130653 11160 maintenance_manager.cc:643] P 3ab52e10dd3d4b95b4fc6922332cf485: LogGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261120 15:01:18.134079 11271 maintenance_manager.cc:419] P 3ab52e10dd3d4b95b4fc6922332cf485: Scheduling UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7): 550 bytes on disk
I20261120 15:01:18.134712 11160 maintenance_manager.cc:643] P 3ab52e10dd3d4b95b4fc6922332cf485: UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20261120 15:01:18.169638 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: CompactRowSetsOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.071s user 0.054s sys 0.000s Metrics: {"bytes_written":136001,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264438,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":95,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1139,"drs_written":1,"lbm_read_time_us":681,"lbm_reads_lt_1ms":46,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":37,"thread_start_us":329,"threads_started":5}
I20261120 15:01:18.170226 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7): 550 bytes on disk
I20261120 15:01:18.170600 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: UndoDeltaBlockGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20261120 15:01:18.672689 11106 maintenance_manager.cc:419] P ce2764a03952415f8a7bf9f1e33b6254: Scheduling LogGCOp(2083630ce86b4ffb80009e7727f50af7): free 920522 bytes of WAL
I20261120 15:01:18.672941 10996 log_reader.cc:385] T 2083630ce86b4ffb80009e7727f50af7: removed 1 log segments from log reader
I20261120 15:01:18.672991 10996 log.cc:1079] T 2083630ce86b4ffb80009e7727f50af7 P ce2764a03952415f8a7bf9f1e33b6254: Deleting log segment in path: /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0/raft_consensus-itest-cluster/ts-3/wal/wals/2083630ce86b4ffb80009e7727f50af7/wal-000000005 (ops 29-35)
I20261120 15:01:18.673291 10996 maintenance_manager.cc:643] P ce2764a03952415f8a7bf9f1e33b6254: LogGCOp(2083630ce86b4ffb80009e7727f50af7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_config_change-itest.cc:500: Failure
Value of: IsRaftConfigMember(added_replica_uuid, cstate.committed_config())
Actual: false
Expected: true
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20261120 15:01:48.272361 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10584
I20261120 15:01:48.278473 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10715
I20261120 15:01:48.285655 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 11141
I20261120 15:01:48.292385 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10977
I20261120 15:01:48.298913 6849 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskkAkmHg/build/debug/bin/kudu with pid 10493
2026-11-20T15:01:48Z chronyd exiting
I20261120 15:01:48.317126 6849 test_util.cc:182] -----------------------------------------------
I20261120 15:01:48.317188 6849 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskkAkmHg/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1795186845751551-6849-0
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap (35324 ms)
[----------] 3 tests from SlowTabletBootstrapTest (47511 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test suites ran. (62556 ms total)
[ PASSED ] 5 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
1 FAILED TEST