[==========] Running 6 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from RaftConfigChangeITest
[ RUN ] RaftConfigChangeITest.TestKudu2147
2026-11-16T21:09:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:04Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20261116 21:09:04.329881 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:45513
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:45513 with env {}
W20261116 21:09:04.406370 479 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:04.406512 479 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:04.406530 479 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:04.407968 479 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:04.408007 479 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:04.408022 479 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:04.408034 479 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:04.409555 479 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:45513
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:04.409708 479 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:04.409833 479 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:04.411983 484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.412020 487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.412037 485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:04.412083 479 server_base.cc:1061] running on GCE node
I20261116 21:09:04.412343 479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:04.412557 479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:04.413733 479 hybrid_clock.cc:648] HybridClock initialized: now 1794863344413718 us; error 45 us; skew 500 ppm
I20261116 21:09:04.414739 479 webserver.cc:577] Webserver started at http://127.0.117.254:46213/ using document root <none> and password file <none>
I20261116 21:09:04.414925 479 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:04.414981 479 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:04.415081 479 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:04.415887 479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data/instance:
uuid: "ced8f38b49b24298ae2c74dc10cdca6c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.416214 479 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal/instance:
uuid: "ced8f38b49b24298ae2c74dc10cdca6c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.417364 479 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20261116 21:09:04.418028 493 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.418197 479 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20261116 21:09:04.418280 479 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal
uuid: "ced8f38b49b24298ae2c74dc10cdca6c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.418354 479 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:04.428185 479 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:04.428447 479 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:04.428576 479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:04.432397 479 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:45513
I20261116 21:09:04.432456 545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:45513 every 8 connection(s)
I20261116 21:09:04.432775 479 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
I20261116 21:09:04.433431 546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:04.433665 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 479
I20261116 21:09:04.433787 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/master-0/wal/instance
I20261116 21:09:04.435726 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c: Bootstrap starting.
I20261116 21:09:04.436296 546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:04.436530 546 log.cc:826] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:04.437142 546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c: No bootstrap required, opened a new log
I20261116 21:09:04.438584 546 raft_consensus.cc:360] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } }
I20261116 21:09:04.438717 546 raft_consensus.cc:386] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:04.438750 546 raft_consensus.cc:748] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced8f38b49b24298ae2c74dc10cdca6c, State: Initialized, Role: FOLLOWER
I20261116 21:09:04.438834 546 consensus_queue.cc:261] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } }
I20261116 21:09:04.438912 546 raft_consensus.cc:400] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:04.438944 546 raft_consensus.cc:499] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:04.438992 546 raft_consensus.cc:3096] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:04.439531 546 raft_consensus.cc:521] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } }
I20261116 21:09:04.439647 546 leader_election.cc:304] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced8f38b49b24298ae2c74dc10cdca6c; no voters:
I20261116 21:09:04.439869 546 leader_election.cc:290] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:04.439911 551 raft_consensus.cc:2826] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:04.440012 551 raft_consensus.cc:705] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [term 1 LEADER]: Becoming Leader. State: Replica: ced8f38b49b24298ae2c74dc10cdca6c, State: Running, Role: LEADER
I20261116 21:09:04.440115 546 sys_catalog.cc:565] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:04.440146 551 consensus_queue.cc:238] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } }
I20261116 21:09:04.440465 551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced8f38b49b24298ae2c74dc10cdca6c. Latest consensus state: current_term: 1 leader_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } } }
I20261116 21:09:04.440498 553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced8f38b49b24298ae2c74dc10cdca6c" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 45513 } } }
I20261116 21:09:04.440533 551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:04.440840 553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:04.441148 559 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:04.441844 559 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:04.443344 559 catalog_manager.cc:1398] Generated new cluster ID: 86afdf8e37aa473383c3a25146de0968
I20261116 21:09:04.443400 559 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:04.454705 559 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:04.455411 559 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:04.476361 559 catalog_manager.cc:6221] T 00000000000000000000000000000000 P ced8f38b49b24298ae2c74dc10cdca6c: Generated new TSK 0
I20261116 21:09:04.476579 559 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:04.482498 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--builtin_ntp_servers=127.0.117.212:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
W20261116 21:09:04.569211 570 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:04.569486 570 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:04.569527 570 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:04.571036 570 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:04.571125 570 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:04.572736 570 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--never_fsync=true
--heap_profile_path=/tmp/kudu.570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:04.572973 570 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:04.573176 570 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:04.575925 578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.575927 576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:04.576162 570 server_base.cc:1061] running on GCE node
W20261116 21:09:04.575929 575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:04.576452 570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:04.576689 570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:04.577853 570 hybrid_clock.cc:648] HybridClock initialized: now 1794863344577843 us; error 27 us; skew 500 ppm
I20261116 21:09:04.578915 570 webserver.cc:577] Webserver started at http://127.0.117.193:45807/ using document root <none> and password file <none>
I20261116 21:09:04.579109 570 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:04.579195 570 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:04.579319 570 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:04.580152 570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data/instance:
uuid: "4edcd7d299d24ab6824c1c423c00246c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.580490 570 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal/instance:
uuid: "4edcd7d299d24ab6824c1c423c00246c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.581655 570 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20261116 21:09:04.582353 584 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.582512 570 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:04.582581 570 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal
uuid: "4edcd7d299d24ab6824c1c423c00246c"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.582652 570 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:04.591436 570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:04.591714 570 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:04.591852 570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:04.592082 570 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:04.592408 570 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:04.592439 570 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.592461 570 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:04.592509 570 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.599749 570 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:34785
I20261116 21:09:04.599782 697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:34785 every 8 connection(s)
I20261116 21:09:04.600112 570 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
I20261116 21:09:04.604777 698 heartbeater.cc:345] Connected to a master server at 127.0.117.254:45513
I20261116 21:09:04.604874 698 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:04.605059 698 heartbeater.cc:508] Master 127.0.117.254:45513 requested a full tablet report, sending...
I20261116 21:09:04.605563 510 ts_manager.cc:210] Registered new tserver with Master: 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193:34785)
I20261116 21:09:04.606211 510 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:39689
I20261116 21:09:04.608603 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 570
I20261116 21:09:04.608686 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal/instance
I20261116 21:09:04.609758 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--builtin_ntp_servers=127.0.117.212:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
W20261116 21:09:04.686455 701 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:04.686640 701 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:04.686681 701 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:04.688230 701 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:04.688324 701 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:04.689898 701 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--never_fsync=true
--heap_profile_path=/tmp/kudu.701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:04.690167 701 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:04.690359 701 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:04.692873 709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.693004 707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.693010 706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:04.693511 701 server_base.cc:1061] running on GCE node
I20261116 21:09:04.693678 701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:04.693902 701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:04.695080 701 hybrid_clock.cc:648] HybridClock initialized: now 1794863344695075 us; error 39 us; skew 500 ppm
I20261116 21:09:04.696262 701 webserver.cc:577] Webserver started at http://127.0.117.194:45947/ using document root <none> and password file <none>
I20261116 21:09:04.696475 701 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:04.696537 701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:04.696659 701 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:04.697561 701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data/instance:
uuid: "94af20118b174cc5a3a2103193edf3b1"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.697898 701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal/instance:
uuid: "94af20118b174cc5a3a2103193edf3b1"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.699039 701 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:04.699834 715 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.700018 701 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:04.700096 701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal
uuid: "94af20118b174cc5a3a2103193edf3b1"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.700152 701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:04.712976 701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:04.713253 701 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:04.713413 701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:04.713639 701 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:04.713958 701 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:04.714010 701 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.714053 701 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:04.714079 701 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.719774 701 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:32871
I20261116 21:09:04.719848 828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:32871 every 8 connection(s)
I20261116 21:09:04.720180 701 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
I20261116 21:09:04.723840 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 701
I20261116 21:09:04.723937 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-1/wal/instance
I20261116 21:09:04.724804 829 heartbeater.cc:345] Connected to a master server at 127.0.117.254:45513
I20261116 21:09:04.724925 829 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:04.725126 829 heartbeater.cc:508] Master 127.0.117.254:45513 requested a full tablet report, sending...
I20261116 21:09:04.725229 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--builtin_ntp_servers=127.0.117.212:46527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000 with env {}
I20261116 21:09:04.725562 510 ts_manager.cc:210] Registered new tserver with Master: 94af20118b174cc5a3a2103193edf3b1 (127.0.117.194:32871)
I20261116 21:09:04.726089 510 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:38893
W20261116 21:09:04.801741 832 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:04.801903 832 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:04.801934 832 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:04.803531 832 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:04.803606 832 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:04.805146 832 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:46527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=3000
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:45513
--never_fsync=true
--heap_profile_path=/tmp/kudu.832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:04.805388 832 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:04.805570 832 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:04.807926 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.807938 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:04.807924 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:04.808292 832 server_base.cc:1061] running on GCE node
I20261116 21:09:04.808471 832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:04.808670 832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:04.809798 832 hybrid_clock.cc:648] HybridClock initialized: now 1794863344809786 us; error 27 us; skew 500 ppm
I20261116 21:09:04.810787 832 webserver.cc:577] Webserver started at http://127.0.117.195:42853/ using document root <none> and password file <none>
I20261116 21:09:04.810995 832 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:04.811055 832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:04.811224 832 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:04.812017 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data/instance:
uuid: "4984ffca67014314a2cf43bab8c5eaee"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.812352 832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal/instance:
uuid: "4984ffca67014314a2cf43bab8c5eaee"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.813484 832 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:04.814121 846 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.814288 832 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:04.814364 832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal
uuid: "4984ffca67014314a2cf43bab8c5eaee"
format_stamp: "Formatted at 2026-11-16 21:09:04 on dist-test-slave-gg0q"
I20261116 21:09:04.814420 832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:04.831058 832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:04.831379 832 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:04.831521 832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:04.831730 832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:04.832056 832 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:04.832110 832 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.832162 832 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:04.832190 832 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:04.838153 832 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:33125
I20261116 21:09:04.838204 959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:33125 every 8 connection(s)
I20261116 21:09:04.838536 832 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
I20261116 21:09:04.839879 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 832
I20261116 21:09:04.839954 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-2/wal/instance
I20261116 21:09:04.843829 960 heartbeater.cc:345] Connected to a master server at 127.0.117.254:45513
I20261116 21:09:04.843936 960 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:04.844231 960 heartbeater.cc:508] Master 127.0.117.254:45513 requested a full tablet report, sending...
I20261116 21:09:04.844678 510 ts_manager.cc:210] Registered new tserver with Master: 4984ffca67014314a2cf43bab8c5eaee (127.0.117.195:33125)
I20261116 21:09:04.845081 510 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:52493
I20261116 21:09:04.852351 471 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20261116 21:09:04.865192 510 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41764:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20261116 21:09:04.865548 510 catalog_manager.cc:7228] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20261116 21:09:04.872208 763 tablet_service.cc:1500] Processing CreateTablet for tablet 9b01af4001d64f3db5f3d72c83608b9c (DEFAULT_TABLE table=test-workload [id=db82d302ad8e4091a1629e1ac603d50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:04.872309 894 tablet_service.cc:1500] Processing CreateTablet for tablet 9b01af4001d64f3db5f3d72c83608b9c (DEFAULT_TABLE table=test-workload [id=db82d302ad8e4091a1629e1ac603d50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:04.872529 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b01af4001d64f3db5f3d72c83608b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:04.872530 763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b01af4001d64f3db5f3d72c83608b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:04.872655 632 tablet_service.cc:1500] Processing CreateTablet for tablet 9b01af4001d64f3db5f3d72c83608b9c (DEFAULT_TABLE table=test-workload [id=db82d302ad8e4091a1629e1ac603d50b]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:04.872932 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b01af4001d64f3db5f3d72c83608b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:04.874941 979 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Bootstrap starting.
I20261116 21:09:04.875547 980 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Bootstrap starting.
I20261116 21:09:04.875653 979 tablet_bootstrap.cc:654] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:04.875947 979 log.cc:826] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:04.876175 980 tablet_bootstrap.cc:654] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:04.876444 980 log.cc:826] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:04.876556 979 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: No bootstrap required, opened a new log
I20261116 21:09:04.876626 979 ts_tablet_manager.cc:1449] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:04.876784 981 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Bootstrap starting.
I20261116 21:09:04.877307 981 tablet_bootstrap.cc:654] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:04.877691 981 log.cc:826] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:04.877883 980 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: No bootstrap required, opened a new log
I20261116 21:09:04.877960 980 ts_tablet_manager.cc:1449] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:04.878286 981 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: No bootstrap required, opened a new log
I20261116 21:09:04.878240 979 raft_consensus.cc:360] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.878367 981 ts_tablet_manager.cc:1449] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20261116 21:09:04.878423 979 raft_consensus.cc:386] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:04.878474 979 raft_consensus.cc:748] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcd7d299d24ab6824c1c423c00246c, State: Initialized, Role: FOLLOWER
I20261116 21:09:04.878603 979 consensus_queue.cc:261] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.879000 979 ts_tablet_manager.cc:1480] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:04.879066 698 heartbeater.cc:500] Master 127.0.117.254:45513 was elected leader, sending a full tablet report...
I20261116 21:09:04.879856 980 raft_consensus.cc:360] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.880013 980 raft_consensus.cc:386] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:04.880069 980 raft_consensus.cc:748] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4984ffca67014314a2cf43bab8c5eaee, State: Initialized, Role: FOLLOWER
I20261116 21:09:04.879987 981 raft_consensus.cc:360] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.880110 981 raft_consensus.cc:386] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:04.880191 981 raft_consensus.cc:748] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94af20118b174cc5a3a2103193edf3b1, State: Initialized, Role: FOLLOWER
I20261116 21:09:04.880231 980 consensus_queue.cc:261] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.880344 981 consensus_queue.cc:261] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:04.880489 980 ts_tablet_manager.cc:1480] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20261116 21:09:04.880558 960 heartbeater.cc:500] Master 127.0.117.254:45513 was elected leader, sending a full tablet report...
I20261116 21:09:04.880666 981 ts_tablet_manager.cc:1480] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261116 21:09:04.880695 829 heartbeater.cc:500] Master 127.0.117.254:45513 was elected leader, sending a full tablet report...
W20261116 21:09:04.970696 830 tablet.cc:2441] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261116 21:09:05.089154 961 tablet.cc:2441] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261116 21:09:05.100895 699 tablet.cc:2441] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 21:09:06.311637 988 raft_consensus.cc:499] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:06.311761 988 raft_consensus.cc:521] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.312126 988 leader_election.cc:290] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4984ffca67014314a2cf43bab8c5eaee (127.0.117.195:33125), 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193:34785)
I20261116 21:09:06.315137 652 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c" candidate_uuid: "94af20118b174cc5a3a2103193edf3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcd7d299d24ab6824c1c423c00246c" is_pre_election: true
I20261116 21:09:06.315142 914 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c" candidate_uuid: "94af20118b174cc5a3a2103193edf3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4984ffca67014314a2cf43bab8c5eaee" is_pre_election: true
I20261116 21:09:06.315271 914 raft_consensus.cc:2490] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af20118b174cc5a3a2103193edf3b1 in term 0.
I20261116 21:09:06.315299 652 raft_consensus.cc:2490] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94af20118b174cc5a3a2103193edf3b1 in term 0.
I20261116 21:09:06.315529 716 leader_election.cc:304] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4984ffca67014314a2cf43bab8c5eaee, 4edcd7d299d24ab6824c1c423c00246c, 94af20118b174cc5a3a2103193edf3b1; no voters:
I20261116 21:09:06.315659 988 raft_consensus.cc:2826] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 21:09:06.315711 988 raft_consensus.cc:499] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:06.315739 988 raft_consensus.cc:3096] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:06.316361 988 raft_consensus.cc:521] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.316511 988 leader_election.cc:290] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [CANDIDATE]: Term 1 election: Requested vote from peers 4984ffca67014314a2cf43bab8c5eaee (127.0.117.195:33125), 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193:34785)
I20261116 21:09:06.316690 652 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c" candidate_uuid: "94af20118b174cc5a3a2103193edf3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcd7d299d24ab6824c1c423c00246c"
I20261116 21:09:06.316691 914 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c" candidate_uuid: "94af20118b174cc5a3a2103193edf3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4984ffca67014314a2cf43bab8c5eaee"
I20261116 21:09:06.316778 914 raft_consensus.cc:3096] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:06.316783 652 raft_consensus.cc:3096] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:06.317546 652 raft_consensus.cc:2490] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af20118b174cc5a3a2103193edf3b1 in term 1.
I20261116 21:09:06.317662 914 raft_consensus.cc:2490] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94af20118b174cc5a3a2103193edf3b1 in term 1.
I20261116 21:09:06.317742 716 leader_election.cc:304] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edcd7d299d24ab6824c1c423c00246c, 94af20118b174cc5a3a2103193edf3b1; no voters:
I20261116 21:09:06.317853 988 raft_consensus.cc:2826] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:06.317971 988 raft_consensus.cc:705] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 94af20118b174cc5a3a2103193edf3b1, State: Running, Role: LEADER
I20261116 21:09:06.318097 988 consensus_queue.cc:238] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.318961 510 catalog_manager.cc:5840] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94af20118b174cc5a3a2103193edf3b1 (127.0.117.194). New cstate: current_term: 1 leader_uuid: "94af20118b174cc5a3a2103193edf3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } health_report { overall_health: HEALTHY } } }
I20261116 21:09:06.319192 995 consensus_queue.cc:1049] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:06.319442 995 consensus_queue.cc:1049] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20261116 21:09:06.583556 692 generic_service.cc:139] {username='slave'} at 127.0.0.1:57118 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261116 21:09:06.583633 692 generic_service.cc:175] {username='slave'} at 127.0.0.1:57118 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261116 21:09:06.584004 692 generic_service.cc:175] {username='slave'} at 127.0.0.1:57118 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
W20261116 21:09:06.584460 823 generic_service.cc:139] {username='slave'} at 127.0.0.1:41468 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261116 21:09:06.584527 823 generic_service.cc:175] {username='slave'} at 127.0.0.1:41468 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261116 21:09:06.584967 823 generic_service.cc:175] {username='slave'} at 127.0.0.1:41468 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '2000'
W20261116 21:09:06.585455 954 generic_service.cc:139] {username='slave'} at 127.0.0.1:48190 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261116 21:09:06.585579 954 generic_service.cc:175] {username='slave'} at 127.0.0.1:48190 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'false'
I20261116 21:09:06.585969 954 generic_service.cc:175] {username='slave'} at 127.0.0.1:48190 changed flags via RPC: heartbeat_inject_latency_before_heartbeat_ms from '0' to '5000'
I20261116 21:09:06.586776 783 consensus_queue.cc:238] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.587306 914 raft_consensus.cc:1285] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 1 FOLLOWER]: Refusing update from remote peer 94af20118b174cc5a3a2103193edf3b1: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20261116 21:09:06.587575 993 consensus_queue.cc:1049] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20261116 21:09:06.588300 995 raft_consensus.cc:2983] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index -1 to 33, VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } }
I20261116 21:09:06.588471 914 raft_consensus.cc:2983] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } }
W20261116 21:09:06.589726 823 generic_service.cc:139] {username='slave'} at 127.0.0.1:41468 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261116 21:09:06.589778 823 generic_service.cc:175] {username='slave'} at 127.0.0.1:41468 changed flags via RPC: follower_reject_update_consensus_requests from 'false' to 'true'
I20261116 21:09:06.590137 914 tablet_service.cc:2033] Received Run Leader Election RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c"
dest_uuid: "4984ffca67014314a2cf43bab8c5eaee"
from {username='slave'} at 127.0.0.1:48190
I20261116 21:09:06.590202 914 raft_consensus.cc:499] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20261116 21:09:06.590224 914 raft_consensus.cc:3096] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:06.590700 914 raft_consensus.cc:521] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.590880 914 leader_election.cc:290] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [CANDIDATE]: Term 2 election: Requested vote from peers 94af20118b174cc5a3a2103193edf3b1 (127.0.117.194:32871)
I20261116 21:09:06.593715 783 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "9b01af4001d64f3db5f3d72c83608b9c" candidate_uuid: "4984ffca67014314a2cf43bab8c5eaee" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: true dest_uuid: "94af20118b174cc5a3a2103193edf3b1"
I20261116 21:09:06.593791 783 raft_consensus.cc:3091] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 LEADER]: Stepping down as leader of term 1
I20261116 21:09:06.593808 783 raft_consensus.cc:748] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 94af20118b174cc5a3a2103193edf3b1, State: Running, Role: LEADER
I20261116 21:09:06.593886 783 consensus_queue.cc:261] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
I20261116 21:09:06.593951 783 raft_consensus.cc:3096] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:06.594516 783 raft_consensus.cc:2490] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4984ffca67014314a2cf43bab8c5eaee in term 2.
I20261116 21:09:06.594727 847 leader_election.cc:304] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4984ffca67014314a2cf43bab8c5eaee, 94af20118b174cc5a3a2103193edf3b1; no voters:
I20261116 21:09:06.594866 997 raft_consensus.cc:2826] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 2 FOLLOWER]: Leader election won for term 2
I20261116 21:09:06.595052 997 raft_consensus.cc:705] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 2 LEADER]: Becoming Leader. State: Replica: 4984ffca67014314a2cf43bab8c5eaee, State: Running, Role: LEADER
I20261116 21:09:06.595201 997 consensus_queue.cc:238] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 1.33, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } }
W20261116 21:09:06.596203 847 consensus_peers.cc:597] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee -> Peer 94af20118b174cc5a3a2103193edf3b1 (127.0.117.194:32871): Couldn't send request to peer 94af20118b174cc5a3a2103193edf3b1. Error code: UNKNOWN_ERROR (1). Status: Illegal state: Rejected: --follower_reject_update_consensus_requests is set to true.. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:08.589900 510 catalog_manager.cc:5840] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 reported cstate change: config changed from index -1 to 33, term changed from 1 to 2, leader changed from 94af20118b174cc5a3a2103193edf3b1 (127.0.117.194) to <none>, VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) evicted. New cstate: current_term: 2 committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } }
I20261116 21:09:08.591431 632 tablet_service.cc:1547] Processing DeleteTablet for tablet 9b01af4001d64f3db5f3d72c83608b9c with delete_type TABLET_DATA_TOMBSTONED (TS 4edcd7d299d24ab6824c1c423c00246c not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:57116
I20261116 21:09:08.591676 1031 tablet_replica.cc:333] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: stopping tablet replica
I20261116 21:09:08.591894 1031 raft_consensus.cc:2261] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 1 FOLLOWER]: Raft consensus shutting down.
I20261116 21:09:08.591988 1031 raft_consensus.cc:2290] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 1 FOLLOWER]: Raft consensus is shut down!
I20261116 21:09:08.592324 1031 ts_tablet_manager.cc:1962] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20261116 21:09:08.593761 1031 ts_tablet_manager.cc:1975] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.32
I20261116 21:09:08.593863 1031 log.cc:1199] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Deleting WAL directory at /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestKudu2147.1794863344313111-471-0/minicluster-data/ts-0/wal/wals/9b01af4001d64f3db5f3d72c83608b9c
I20261116 21:09:08.594424 494 catalog_manager.cc:5065] TS 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193:34785): tablet 9b01af4001d64f3db5f3d72c83608b9c (table test-workload [id=db82d302ad8e4091a1629e1ac603d50b]) successfully deleted
W20261116 21:09:08.624361 823 generic_service.cc:139] {username='slave'} at 127.0.0.1:41468 forcing change of non-runtime-safe flag follower_reject_update_consensus_requests
I20261116 21:09:08.624446 823 generic_service.cc:175] {username='slave'} at 127.0.0.1:41468 changed flags via RPC: follower_reject_update_consensus_requests from 'true' to 'false'
I20261116 21:09:08.626049 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 1.33 2.34
I20261116 21:09:08.727912 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 1.33 2.34
I20261116 21:09:08.929757 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 1.33 2.34
I20261116 21:09:09.231360 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 1.33 2.34
I20261116 21:09:09.633361 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.34 2.34
I20261116 21:09:10.135169 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.34 2.34
I20261116 21:09:10.592020 510 catalog_manager.cc:5840] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 reported cstate change: leader changed from <none> to 4984ffca67014314a2cf43bab8c5eaee (127.0.117.195). New cstate: current_term: 2 leader_uuid: "4984ffca67014314a2cf43bab8c5eaee" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } }
I20261116 21:09:10.736912 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.34 2.34
I20261116 21:09:11.438539 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.34 2.34
I20261116 21:09:11.590755 914 consensus_queue.cc:238] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 33, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } }
I20261116 21:09:11.591774 783 raft_consensus.cc:1285] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 2 FOLLOWER]: Refusing update from remote peer 4984ffca67014314a2cf43bab8c5eaee: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20261116 21:09:11.592082 1045 consensus_queue.cc:1049] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [LEADER]: Connected to new peer: Peer: permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20261116 21:09:11.593113 1044 raft_consensus.cc:2983] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } } }
I20261116 21:09:11.593261 783 raft_consensus.cc:2983] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 33 to 35, NON_VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) added. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } } }
I20261116 21:09:11.594285 496 catalog_manager.cc:5247] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b01af4001d64f3db5f3d72c83608b9c with cas_config_opid_index 33: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20261116 21:09:11.594717 847 consensus_peers.cc:597] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee -> Peer 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193:34785): Couldn't send request to peer 4edcd7d299d24ab6824c1c423c00246c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:12.240389 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.35 2.35
I20261116 21:09:12.594317 510 catalog_manager.cc:5840] T 9b01af4001d64f3db5f3d72c83608b9c P 94af20118b174cc5a3a2103193edf3b1 reported cstate change: config changed from index 33 to 35, NON_VOTER 4edcd7d299d24ab6824c1c423c00246c (127.0.117.193) added. New cstate: current_term: 2 leader_uuid: "4984ffca67014314a2cf43bab8c5eaee" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } } }
I20261116 21:09:13.142117 471 cluster_itest_util.cc:258] Not converged past 1 yet: <uninitialized op> 2.35 2.35
I20261116 21:09:14.096513 1055 ts_tablet_manager.cc:979] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Initiating tablet copy from peer 4984ffca67014314a2cf43bab8c5eaee (127.0.117.195:33125)
I20261116 21:09:14.096761 1055 tablet_copy_client.cc:323] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.117.195:33125
I20261116 21:09:14.099891 934 tablet_copy_service.cc:140] P 4984ffca67014314a2cf43bab8c5eaee: Received BeginTabletCopySession request for tablet 9b01af4001d64f3db5f3d72c83608b9c from peer 4edcd7d299d24ab6824c1c423c00246c ({username='slave'} at 127.0.117.193:49147)
I20261116 21:09:14.099975 934 tablet_copy_service.cc:161] P 4984ffca67014314a2cf43bab8c5eaee: Beginning new tablet copy session on tablet 9b01af4001d64f3db5f3d72c83608b9c from peer 4edcd7d299d24ab6824c1c423c00246c at {username='slave'} at 127.0.117.193:49147: session id = 4edcd7d299d24ab6824c1c423c00246c-9b01af4001d64f3db5f3d72c83608b9c
I20261116 21:09:14.100595 934 tablet_copy_source_session.cc:215] T 9b01af4001d64f3db5f3d72c83608b9c P 4984ffca67014314a2cf43bab8c5eaee: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 21:09:14.101263 1055 ts_tablet_manager.cc:1962] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20261116 21:09:14.102943 1055 ts_tablet_manager.cc:1975] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.32
I20261116 21:09:14.103091 1055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b01af4001d64f3db5f3d72c83608b9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:14.103834 1055 tablet_copy_client.cc:806] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet copy: Starting download of 0 data blocks...
I20261116 21:09:14.103972 1055 tablet_copy_client.cc:670] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet copy: Starting download of 1 WAL segments...
I20261116 21:09:14.105103 1055 tablet_copy_client.cc:538] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 21:09:14.105770 1055 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Bootstrap starting.
I20261116 21:09:14.112492 1055 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Bootstrap replayed 1/1 log segments. Stats: ops{read=35 overwritten=0 applied=35 ignored=0} inserts{seen=1550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 21:09:14.112728 1055 tablet_bootstrap.cc:492] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Bootstrap complete.
I20261116 21:09:14.112859 1055 ts_tablet_manager.cc:1449] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20261116 21:09:14.113029 1055 raft_consensus.cc:360] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } }
I20261116 21:09:14.113106 1055 raft_consensus.cc:748] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4edcd7d299d24ab6824c1c423c00246c, State: Initialized, Role: LEARNER
I20261116 21:09:14.113145 1055 consensus_queue.cc:261] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 35, Last appended: 2.35, Last appended by leader: 35, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4984ffca67014314a2cf43bab8c5eaee" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 33125 } } peers { permanent_uuid: "94af20118b174cc5a3a2103193edf3b1" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 32871 } } peers { permanent_uuid: "4edcd7d299d24ab6824c1c423c00246c" member_type: NON_VOTER last_known_addr { host: "127.0.117.193" port: 34785 } attrs { promote: true } }
I20261116 21:09:14.113365 1055 ts_tablet_manager.cc:1480] T 9b01af4001d64f3db5f3d72c83608b9c P 4edcd7d299d24ab6824c1c423c00246c: Time spent starting tablet: real 0.000s user 0.003s sys 0.001s
I20261116 21:09:14.113693 934 tablet_copy_service.cc:342] P 4984ffca67014314a2cf43bab8c5eaee: Request end of tablet copy session 4edcd7d299d24ab6824c1c423c00246c-9b01af4001d64f3db5f3d72c83608b9c received from {username='slave'} at 127.0.117.193:49147
I20261116 21:09:14.113759 934 tablet_copy_service.cc:434] P 4984ffca67014314a2cf43bab8c5eaee: ending tablet copy session 4edcd7d299d24ab6824c1c423c00246c-9b01af4001d64f3db5f3d72c83608b9c on tablet 9b01af4001d64f3db5f3d72c83608b9c with peer 4edcd7d299d24ab6824c1c423c00246c
I20261116 21:09:14.144021 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 570
I20261116 21:09:14.149834 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 701
I20261116 21:09:14.155226 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 832
I20261116 21:09:14.160204 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 479
2026-11-16T21:09:14Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestKudu2147 (9862 ms)
[ RUN ] RaftConfigChangeITest.TestNonVoterPromotion
2026-11-16T21:09:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:14Z Disabled control of system clock
I20261116 21:09:14.187829 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:40665
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:40665
--raft_prepare_replacement_before_eviction=true
--catalog_manager_evict_excess_replicas=false with env {}
W20261116 21:09:14.265556 1064 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:14.265694 1064 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:14.265713 1064 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:14.267273 1064 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:14.267318 1064 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:14.267331 1064 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:14.267344 1064 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:14.268970 1064 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal
--catalog_manager_evict_excess_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:40665
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:14.269181 1064 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:14.269402 1064 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:14.272526 1070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.272586 1072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.272651 1069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.272657 1064 server_base.cc:1061] running on GCE node
I20261116 21:09:14.272938 1064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:14.273159 1064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:14.274300 1064 hybrid_clock.cc:648] HybridClock initialized: now 1794863354274274 us; error 41 us; skew 500 ppm
I20261116 21:09:14.275345 1064 webserver.cc:577] Webserver started at http://127.0.117.254:36795/ using document root <none> and password file <none>
I20261116 21:09:14.275579 1064 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:14.275624 1064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:14.275734 1064 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:14.276575 1064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data/instance:
uuid: "94b4b0d94d5548df91fd32634e5f97fc"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.276916 1064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal/instance:
uuid: "94b4b0d94d5548df91fd32634e5f97fc"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.278213 1064 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20261116 21:09:14.279043 1078 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.279235 1064 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:14.279321 1064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal
uuid: "94b4b0d94d5548df91fd32634e5f97fc"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.279382 1064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:14.295018 1064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:14.295351 1064 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:14.295511 1064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:14.299924 1064 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:40665
I20261116 21:09:14.300000 1130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:40665 every 8 connection(s)
I20261116 21:09:14.300326 1064 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
I20261116 21:09:14.300937 1131 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:14.301873 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1064
I20261116 21:09:14.301997 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/master-0/wal/instance
I20261116 21:09:14.303365 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc: Bootstrap starting.
I20261116 21:09:14.304070 1131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:14.304363 1131 log.cc:826] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:14.304939 1131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc: No bootstrap required, opened a new log
I20261116 21:09:14.306466 1131 raft_consensus.cc:360] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } }
I20261116 21:09:14.306579 1131 raft_consensus.cc:386] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:14.306598 1131 raft_consensus.cc:748] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b4b0d94d5548df91fd32634e5f97fc, State: Initialized, Role: FOLLOWER
I20261116 21:09:14.306713 1131 consensus_queue.cc:261] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } }
I20261116 21:09:14.306780 1131 raft_consensus.cc:400] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:14.306834 1131 raft_consensus.cc:499] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:14.306880 1131 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:14.307420 1131 raft_consensus.cc:521] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } }
I20261116 21:09:14.307514 1131 leader_election.cc:304] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b4b0d94d5548df91fd32634e5f97fc; no voters:
I20261116 21:09:14.307698 1131 leader_election.cc:290] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:14.307816 1136 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:14.307924 1131 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:14.307976 1136 raft_consensus.cc:705] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [term 1 LEADER]: Becoming Leader. State: Replica: 94b4b0d94d5548df91fd32634e5f97fc, State: Running, Role: LEADER
I20261116 21:09:14.308110 1136 consensus_queue.cc:238] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } }
I20261116 21:09:14.308550 1138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b4b0d94d5548df91fd32634e5f97fc. Latest consensus state: current_term: 1 leader_uuid: "94b4b0d94d5548df91fd32634e5f97fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } } }
I20261116 21:09:14.308616 1137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b4b0d94d5548df91fd32634e5f97fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b4b0d94d5548df91fd32634e5f97fc" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 40665 } } }
I20261116 21:09:14.308660 1138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:14.308686 1137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:14.309173 1145 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:14.309711 1145 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:14.311158 1145 catalog_manager.cc:1398] Generated new cluster ID: b93820d50019438a89d4a97e4d7cc210
I20261116 21:09:14.311213 1145 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:14.331027 1145 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:14.331581 1145 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:14.341708 1145 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 94b4b0d94d5548df91fd32634e5f97fc: Generated new TSK 0
I20261116 21:09:14.341920 1145 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:14.350101 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--builtin_ntp_servers=127.0.117.212:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:14.449156 1155 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:14.449366 1155 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:14.449416 1155 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:14.451009 1155 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:14.451116 1155 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:14.452688 1155 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.1155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:14.452919 1155 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:14.453115 1155 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:14.455657 1160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.455688 1161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.455840 1163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.456045 1155 server_base.cc:1061] running on GCE node
I20261116 21:09:14.456216 1155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:14.456408 1155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:14.457557 1155 hybrid_clock.cc:648] HybridClock initialized: now 1794863354457542 us; error 31 us; skew 500 ppm
I20261116 21:09:14.458666 1155 webserver.cc:577] Webserver started at http://127.0.117.193:38305/ using document root <none> and password file <none>
I20261116 21:09:14.458884 1155 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:14.458943 1155 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:14.459046 1155 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:14.459911 1155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data/instance:
uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.460247 1155 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal/instance:
uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.461432 1155 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20261116 21:09:14.462167 1169 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.462338 1155 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:14.462404 1155 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal
uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.462461 1155 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:14.481178 1155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:14.481483 1155 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:14.481621 1155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:14.481864 1155 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:14.482223 1155 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:14.482254 1155 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.482277 1155 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:14.482326 1155 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.489578 1155 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:33157
I20261116 21:09:14.489637 1282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:33157 every 8 connection(s)
I20261116 21:09:14.489948 1155 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
I20261116 21:09:14.494468 1283 heartbeater.cc:345] Connected to a master server at 127.0.117.254:40665
I20261116 21:09:14.494557 1283 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:14.494755 1283 heartbeater.cc:508] Master 127.0.117.254:40665 requested a full tablet report, sending...
I20261116 21:09:14.495270 1095 ts_manager.cc:210] Registered new tserver with Master: 4f055ea4fcd6418fa3eda3b407b3b0a2 (127.0.117.193:33157)
I20261116 21:09:14.495765 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1155
I20261116 21:09:14.495839 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-0/wal/instance
I20261116 21:09:14.495970 1095 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:50319
I20261116 21:09:14.496948 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--builtin_ntp_servers=127.0.117.212:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:14.583376 1286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:14.583515 1286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:14.583534 1286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:14.585060 1286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:14.585116 1286 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:14.586745 1286 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.1286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:14.586951 1286 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:14.587155 1286 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:14.589607 1294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.589607 1291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.589635 1292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.589906 1286 server_base.cc:1061] running on GCE node
I20261116 21:09:14.590091 1286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:14.590287 1286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:14.591452 1286 hybrid_clock.cc:648] HybridClock initialized: now 1794863354591440 us; error 23 us; skew 500 ppm
I20261116 21:09:14.592516 1286 webserver.cc:577] Webserver started at http://127.0.117.194:45673/ using document root <none> and password file <none>
I20261116 21:09:14.592730 1286 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:14.592792 1286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:14.592896 1286 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:14.593770 1286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data/instance:
uuid: "58016a967bb04f44a797effd919da0a0"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.594107 1286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal/instance:
uuid: "58016a967bb04f44a797effd919da0a0"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.595301 1286 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:14.596060 1300 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.596246 1286 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:14.596312 1286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal
uuid: "58016a967bb04f44a797effd919da0a0"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.596355 1286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:14.615994 1286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:14.616247 1286 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:14.616343 1286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:14.616511 1286 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:14.616873 1286 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:14.616907 1286 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.616930 1286 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:14.616984 1286 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.622852 1286 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:37207
I20261116 21:09:14.622896 1413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:37207 every 8 connection(s)
I20261116 21:09:14.623201 1286 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
I20261116 21:09:14.627560 1414 heartbeater.cc:345] Connected to a master server at 127.0.117.254:40665
I20261116 21:09:14.627688 1414 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:14.627884 1414 heartbeater.cc:508] Master 127.0.117.254:40665 requested a full tablet report, sending...
I20261116 21:09:14.628266 1095 ts_manager.cc:210] Registered new tserver with Master: 58016a967bb04f44a797effd919da0a0 (127.0.117.194:37207)
I20261116 21:09:14.628614 1095 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:44419
I20261116 21:09:14.631646 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1286
I20261116 21:09:14.631724 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-1/wal/instance
I20261116 21:09:14.632966 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--builtin_ntp_servers=127.0.117.212:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:14.712231 1417 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:14.712385 1417 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:14.712405 1417 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:14.713897 1417 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:14.713948 1417 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:14.715850 1417 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.1417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:14.716116 1417 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:14.716313 1417 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:14.718963 1423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.718958 1422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.718976 1417 server_base.cc:1061] running on GCE node
W20261116 21:09:14.718958 1425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.719377 1417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:14.719545 1417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:14.720736 1417 hybrid_clock.cc:648] HybridClock initialized: now 1794863354720721 us; error 26 us; skew 500 ppm
I20261116 21:09:14.722218 1417 webserver.cc:577] Webserver started at http://127.0.117.195:42705/ using document root <none> and password file <none>
I20261116 21:09:14.722473 1417 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:14.722528 1417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:14.722628 1417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:14.723428 1417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data/instance:
uuid: "776e093059734d3ea85c7d03c43c99df"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.723707 1417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal/instance:
uuid: "776e093059734d3ea85c7d03c43c99df"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.724854 1417 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20261116 21:09:14.725641 1431 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.725839 1417 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:14.725919 1417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal
uuid: "776e093059734d3ea85c7d03c43c99df"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.725993 1417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:14.750546 1417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:14.750859 1417 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:14.751000 1417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:14.751238 1417 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:14.751605 1417 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:14.751657 1417 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.751700 1417 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:14.751730 1417 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.757596 1417 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:38487
I20261116 21:09:14.757674 1544 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:38487 every 8 connection(s)
I20261116 21:09:14.758037 1417 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
I20261116 21:09:14.762758 1545 heartbeater.cc:345] Connected to a master server at 127.0.117.254:40665
I20261116 21:09:14.762869 1545 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:14.763087 1545 heartbeater.cc:508] Master 127.0.117.254:40665 requested a full tablet report, sending...
I20261116 21:09:14.763528 1095 ts_manager.cc:210] Registered new tserver with Master: 776e093059734d3ea85c7d03c43c99df (127.0.117.195:38487)
I20261116 21:09:14.763924 1095 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:52331
I20261116 21:09:14.767732 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1417
I20261116 21:09:14.767819 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-2/wal/instance
I20261116 21:09:14.769181 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.196:0
--local_ip_for_outbound_sockets=127.0.117.196
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--builtin_ntp_servers=127.0.117.212:36111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:14.851472 1548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:14.851630 1548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:14.851650 1548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:14.853209 1548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:14.853266 1548 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.196
I20261116 21:09:14.854997 1548 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:36111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:40665
--never_fsync=true
--heap_profile_path=/tmp/kudu.1548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.196
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:14.855207 1548 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:14.855396 1548 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:14.857966 1556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.857986 1553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:14.858009 1554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:14.858325 1548 server_base.cc:1061] running on GCE node
I20261116 21:09:14.858512 1548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:14.858736 1548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:14.859896 1548 hybrid_clock.cc:648] HybridClock initialized: now 1794863354859882 us; error 43 us; skew 500 ppm
I20261116 21:09:14.861003 1548 webserver.cc:577] Webserver started at http://127.0.117.196:44737/ using document root <none> and password file <none>
I20261116 21:09:14.861215 1548 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:14.861279 1548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:14.861410 1548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:14.862252 1548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data/instance:
uuid: "f66005c75c374cae9d89302c00ff0a1c"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.862617 1548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal/instance:
uuid: "f66005c75c374cae9d89302c00ff0a1c"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.863864 1548 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:14.864694 1562 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.864970 1548 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:14.865051 1548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal
uuid: "f66005c75c374cae9d89302c00ff0a1c"
format_stamp: "Formatted at 2026-11-16 21:09:14 on dist-test-slave-gg0q"
I20261116 21:09:14.865126 1548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:14.905929 1548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:14.906250 1548 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:14.906394 1548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:14.906620 1548 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:14.906970 1548 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:14.907027 1548 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.907070 1548 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:14.907099 1548 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:14.913091 1548 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.196:34967
I20261116 21:09:14.913139 1675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.196:34967 every 8 connection(s)
I20261116 21:09:14.913524 1548 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
I20261116 21:09:14.914525 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1548
I20261116 21:09:14.914613 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestNonVoterPromotion.1794863344313111-471-0/minicluster-data/ts-3/wal/instance
I20261116 21:09:14.918051 1676 heartbeater.cc:345] Connected to a master server at 127.0.117.254:40665
I20261116 21:09:14.918144 1676 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:14.918299 1676 heartbeater.cc:508] Master 127.0.117.254:40665 requested a full tablet report, sending...
I20261116 21:09:14.918672 1095 ts_manager.cc:210] Registered new tserver with Master: f66005c75c374cae9d89302c00ff0a1c (127.0.117.196:34967)
I20261116 21:09:14.919056 1095 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.196:39025
I20261116 21:09:14.926798 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:14.939862 1095 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50942:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261116 21:09:14.946941 1217 tablet_service.cc:1500] Processing CreateTablet for tablet 7c13e6b1c3064d85923eebcd00c43c07 (DEFAULT_TABLE table=test-workload [id=5383ebecfdc94f09acc73920bbc5747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:14.947276 1217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c13e6b1c3064d85923eebcd00c43c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:14.947520 1348 tablet_service.cc:1500] Processing CreateTablet for tablet 7c13e6b1c3064d85923eebcd00c43c07 (DEFAULT_TABLE table=test-workload [id=5383ebecfdc94f09acc73920bbc5747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:14.947752 1348 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c13e6b1c3064d85923eebcd00c43c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:14.947741 1479 tablet_service.cc:1500] Processing CreateTablet for tablet 7c13e6b1c3064d85923eebcd00c43c07 (DEFAULT_TABLE table=test-workload [id=5383ebecfdc94f09acc73920bbc5747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:14.947952 1479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c13e6b1c3064d85923eebcd00c43c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:14.950078 1698 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Bootstrap starting.
I20261116 21:09:14.950312 1700 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Bootstrap starting.
I20261116 21:09:14.950368 1699 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Bootstrap starting.
I20261116 21:09:14.950788 1698 tablet_bootstrap.cc:654] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:14.950961 1700 tablet_bootstrap.cc:654] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:14.950973 1699 tablet_bootstrap.cc:654] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:14.951064 1698 log.cc:826] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:14.951251 1699 log.cc:826] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:14.951253 1700 log.cc:826] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:14.951821 1698 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: No bootstrap required, opened a new log
I20261116 21:09:14.951840 1700 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: No bootstrap required, opened a new log
I20261116 21:09:14.951875 1698 ts_tablet_manager.cc:1449] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:14.951918 1700 ts_tablet_manager.cc:1449] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:14.953101 1700 raft_consensus.cc:360] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.953150 1698 raft_consensus.cc:360] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.953240 1698 raft_consensus.cc:386] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:14.953258 1698 raft_consensus.cc:748] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776e093059734d3ea85c7d03c43c99df, State: Initialized, Role: FOLLOWER
I20261116 21:09:14.953394 1698 consensus_queue.cc:261] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.953229 1700 raft_consensus.cc:386] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:14.953655 1700 raft_consensus.cc:748] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58016a967bb04f44a797effd919da0a0, State: Initialized, Role: FOLLOWER
I20261116 21:09:14.953673 1698 ts_tablet_manager.cc:1480] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:14.953750 1700 consensus_queue.cc:261] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.954022 1700 ts_tablet_manager.cc:1480] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261116 21:09:14.954065 1414 heartbeater.cc:500] Master 127.0.117.254:40665 was elected leader, sending a full tablet report...
I20261116 21:09:14.954118 1699 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: No bootstrap required, opened a new log
I20261116 21:09:14.954128 1545 heartbeater.cc:500] Master 127.0.117.254:40665 was elected leader, sending a full tablet report...
I20261116 21:09:14.954178 1699 ts_tablet_manager.cc:1449] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20261116 21:09:14.955731 1699 raft_consensus.cc:360] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.955880 1699 raft_consensus.cc:386] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:14.955924 1699 raft_consensus.cc:748] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f055ea4fcd6418fa3eda3b407b3b0a2, State: Initialized, Role: FOLLOWER
I20261116 21:09:14.955997 1699 consensus_queue.cc:261] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:14.956233 1699 ts_tablet_manager.cc:1480] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:14.956305 1283 heartbeater.cc:500] Master 127.0.117.254:40665 was elected leader, sending a full tablet report...
W20261116 21:09:14.990648 1284 tablet.cc:2441] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261116 21:09:15.008563 1546 tablet.cc:2441] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 21:09:15.029143 1705 raft_consensus.cc:499] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:15.029306 1705 raft_consensus.cc:521] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:15.029764 1705 leader_election.cc:290] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f055ea4fcd6418fa3eda3b407b3b0a2 (127.0.117.193:33157), 776e093059734d3ea85c7d03c43c99df (127.0.117.195:38487)
I20261116 21:09:15.032631 1237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "7c13e6b1c3064d85923eebcd00c43c07" candidate_uuid: "58016a967bb04f44a797effd919da0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" is_pre_election: true
I20261116 21:09:15.032788 1237 raft_consensus.cc:2490] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58016a967bb04f44a797effd919da0a0 in term 0.
I20261116 21:09:15.033020 1302 leader_election.cc:304] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f055ea4fcd6418fa3eda3b407b3b0a2, 58016a967bb04f44a797effd919da0a0; no voters:
I20261116 21:09:15.033161 1499 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "7c13e6b1c3064d85923eebcd00c43c07" candidate_uuid: "58016a967bb04f44a797effd919da0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776e093059734d3ea85c7d03c43c99df" is_pre_election: true
I20261116 21:09:15.033254 1705 raft_consensus.cc:2826] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 21:09:15.033355 1705 raft_consensus.cc:499] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:15.033354 1499 raft_consensus.cc:2490] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58016a967bb04f44a797effd919da0a0 in term 0.
I20261116 21:09:15.033383 1705 raft_consensus.cc:3096] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:15.034101 1705 raft_consensus.cc:521] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:15.034214 1705 leader_election.cc:290] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [CANDIDATE]: Term 1 election: Requested vote from peers 4f055ea4fcd6418fa3eda3b407b3b0a2 (127.0.117.193:33157), 776e093059734d3ea85c7d03c43c99df (127.0.117.195:38487)
I20261116 21:09:15.034423 1237 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "7c13e6b1c3064d85923eebcd00c43c07" candidate_uuid: "58016a967bb04f44a797effd919da0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2"
I20261116 21:09:15.034507 1237 raft_consensus.cc:3096] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:15.034534 1499 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "7c13e6b1c3064d85923eebcd00c43c07" candidate_uuid: "58016a967bb04f44a797effd919da0a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776e093059734d3ea85c7d03c43c99df"
I20261116 21:09:15.034626 1499 raft_consensus.cc:3096] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:15.035128 1237 raft_consensus.cc:2490] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58016a967bb04f44a797effd919da0a0 in term 1.
I20261116 21:09:15.035287 1302 leader_election.cc:304] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f055ea4fcd6418fa3eda3b407b3b0a2, 58016a967bb04f44a797effd919da0a0; no voters:
I20261116 21:09:15.035389 1705 raft_consensus.cc:2826] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:15.035568 1705 raft_consensus.cc:705] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 1 LEADER]: Becoming Leader. State: Replica: 58016a967bb04f44a797effd919da0a0, State: Running, Role: LEADER
I20261116 21:09:15.035681 1705 consensus_queue.cc:238] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } }
I20261116 21:09:15.035832 1499 raft_consensus.cc:2490] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58016a967bb04f44a797effd919da0a0 in term 1.
I20261116 21:09:15.036520 1095 catalog_manager.cc:5840] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58016a967bb04f44a797effd919da0a0 (127.0.117.194). New cstate: current_term: 1 leader_uuid: "58016a967bb04f44a797effd919da0a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } health_report { overall_health: UNKNOWN } } }
I20261116 21:09:15.036737 1710 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:15.036911 1710 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:15.065853 1368 consensus_queue.cc:238] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } }
I20261116 21:09:15.066771 1237 raft_consensus.cc:1285] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 1 FOLLOWER]: Refusing update from remote peer 58016a967bb04f44a797effd919da0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261116 21:09:15.066776 1499 raft_consensus.cc:1285] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 1 FOLLOWER]: Refusing update from remote peer 58016a967bb04f44a797effd919da0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20261116 21:09:15.067005 1705 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 21:09:15.067181 1710 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20261116 21:09:15.068030 1705 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } } }
I20261116 21:09:15.068194 1237 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } } }
I20261116 21:09:15.069548 1499 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } } }
I20261116 21:09:15.069607 1095 catalog_manager.cc:5840] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 reported cstate change: config changed from index -1 to 2, NON_VOTER f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) added. New cstate: current_term: 1 leader_uuid: "58016a967bb04f44a797effd919da0a0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20261116 21:09:15.071542 1302 consensus_peers.cc:597] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 -> Peer f66005c75c374cae9d89302c00ff0a1c (127.0.117.196:34967): Couldn't send request to peer f66005c75c374cae9d89302c00ff0a1c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c13e6b1c3064d85923eebcd00c43c07. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:15.555994 1718 ts_tablet_manager.cc:979] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Initiating tablet copy from peer 58016a967bb04f44a797effd919da0a0 (127.0.117.194:37207)
I20261116 21:09:15.556258 1718 tablet_copy_client.cc:323] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.117.194:37207
I20261116 21:09:15.559195 1388 tablet_copy_service.cc:140] P 58016a967bb04f44a797effd919da0a0: Received BeginTabletCopySession request for tablet 7c13e6b1c3064d85923eebcd00c43c07 from peer f66005c75c374cae9d89302c00ff0a1c ({username='slave'} at 127.0.117.196:33327)
I20261116 21:09:15.559279 1388 tablet_copy_service.cc:161] P 58016a967bb04f44a797effd919da0a0: Beginning new tablet copy session on tablet 7c13e6b1c3064d85923eebcd00c43c07 from peer f66005c75c374cae9d89302c00ff0a1c at {username='slave'} at 127.0.117.196:33327: session id = f66005c75c374cae9d89302c00ff0a1c-7c13e6b1c3064d85923eebcd00c43c07
I20261116 21:09:15.559809 1388 tablet_copy_source_session.cc:215] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: Tablet Copy: opened 0 blocks and 1 log segments
I20261116 21:09:15.560575 1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c13e6b1c3064d85923eebcd00c43c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:15.562551 1718 tablet_copy_client.cc:806] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: tablet copy: Starting download of 0 data blocks...
I20261116 21:09:15.562718 1718 tablet_copy_client.cc:670] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: tablet copy: Starting download of 1 WAL segments...
I20261116 21:09:15.563604 1718 tablet_copy_client.cc:538] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 21:09:15.564280 1718 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Bootstrap starting.
I20261116 21:09:15.565567 1718 log.cc:826] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:15.566411 1718 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 21:09:15.566618 1718 tablet_bootstrap.cc:492] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Bootstrap complete.
I20261116 21:09:15.566735 1718 ts_tablet_manager.cc:1449] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:15.567874 1718 raft_consensus.cc:360] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } }
I20261116 21:09:15.567994 1718 raft_consensus.cc:748] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f66005c75c374cae9d89302c00ff0a1c, State: Initialized, Role: LEARNER
I20261116 21:09:15.568073 1718 consensus_queue.cc:261] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: true } }
I20261116 21:09:15.568336 1718 ts_tablet_manager.cc:1480] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:15.568535 1676 heartbeater.cc:500] Master 127.0.117.254:40665 was elected leader, sending a full tablet report...
I20261116 21:09:15.568754 1388 tablet_copy_service.cc:342] P 58016a967bb04f44a797effd919da0a0: Request end of tablet copy session f66005c75c374cae9d89302c00ff0a1c-7c13e6b1c3064d85923eebcd00c43c07 received from {username='slave'} at 127.0.117.196:33327
I20261116 21:09:15.568820 1388 tablet_copy_service.cc:434] P 58016a967bb04f44a797effd919da0a0: ending tablet copy session f66005c75c374cae9d89302c00ff0a1c-7c13e6b1c3064d85923eebcd00c43c07 on tablet 7c13e6b1c3064d85923eebcd00c43c07 with peer f66005c75c374cae9d89302c00ff0a1c
I20261116 21:09:15.950263 1630 raft_consensus.cc:1227] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20261116 21:09:16.357682 1710 raft_consensus.cc:1074] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0: attempting to promote NON_VOTER f66005c75c374cae9d89302c00ff0a1c to VOTER
I20261116 21:09:16.357947 1710 consensus_queue.cc:238] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } }
I20261116 21:09:16.358666 1630 raft_consensus.cc:1285] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [term 1 LEARNER]: Refusing update from remote peer 58016a967bb04f44a797effd919da0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261116 21:09:16.358691 1237 raft_consensus.cc:1285] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 1 FOLLOWER]: Refusing update from remote peer 58016a967bb04f44a797effd919da0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261116 21:09:16.358691 1499 raft_consensus.cc:1285] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 1 FOLLOWER]: Refusing update from remote peer 58016a967bb04f44a797effd919da0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20261116 21:09:16.358948 1726 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 21:09:16.359087 1710 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 21:09:16.359293 1725 consensus_queue.cc:1049] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20261116 21:09:16.360003 1710 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } } }
I20261116 21:09:16.360152 1237 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 4f055ea4fcd6418fa3eda3b407b3b0a2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } } }
I20261116 21:09:16.360368 1630 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P f66005c75c374cae9d89302c00ff0a1c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } } }
I20261116 21:09:16.361639 1094 catalog_manager.cc:5840] T 7c13e6b1c3064d85923eebcd00c43c07 P 58016a967bb04f44a797effd919da0a0 reported cstate change: config changed from index 2 to 3, f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58016a967bb04f44a797effd919da0a0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20261116 21:09:16.361676 1499 raft_consensus.cc:2983] T 7c13e6b1c3064d85923eebcd00c43c07 P 776e093059734d3ea85c7d03c43c99df [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f66005c75c374cae9d89302c00ff0a1c (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58016a967bb04f44a797effd919da0a0" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 37207 } } peers { permanent_uuid: "4f055ea4fcd6418fa3eda3b407b3b0a2" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 33157 } } peers { permanent_uuid: "776e093059734d3ea85c7d03c43c99df" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 38487 } } peers { permanent_uuid: "f66005c75c374cae9d89302c00ff0a1c" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 34967 } attrs { promote: false } } }
I20261116 21:09:16.362843 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1155
I20261116 21:09:16.369405 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1286
W20261116 21:09:16.369424 1082 connection.cc:466] server connection from 127.0.117.193:50319 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.117.193:50319 (request call id 8) could send its response
W20261116 21:09:16.374658 1080 connection.cc:573] server connection from 127.0.117.194:44419 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
I20261116 21:09:16.374953 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1417
I20261116 21:09:16.379805 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1548
I20261116 21:09:16.384837 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1064
2026-11-16T21:09:16Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestNonVoterPromotion (2224 ms)
[ RUN ] RaftConfigChangeITest.TestBulkChangeConfig
2026-11-16T21:09:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:16Z Disabled control of system clock
I20261116 21:09:16.412533 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:35923
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:35923
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:16.490243 1739 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:16.490389 1739 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:16.490407 1739 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:16.491873 1739 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:16.491914 1739 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:16.491927 1739 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:16.491940 1739 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:16.493494 1739 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:35923
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:35923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:16.493652 1739 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:16.493780 1739 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:16.495945 1747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:16.496104 1739 server_base.cc:1061] running on GCE node
W20261116 21:09:16.495985 1745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.495954 1744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:16.496476 1739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:16.496744 1739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:16.497918 1739 hybrid_clock.cc:648] HybridClock initialized: now 1794863356497895 us; error 36 us; skew 500 ppm
I20261116 21:09:16.498967 1739 webserver.cc:577] Webserver started at http://127.0.117.254:37937/ using document root <none> and password file <none>
I20261116 21:09:16.499218 1739 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:16.499284 1739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:16.499387 1739 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:16.500217 1739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data/instance:
uuid: "180facb734604feeabf76f00299f0451"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.500547 1739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal/instance:
uuid: "180facb734604feeabf76f00299f0451"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.501816 1739 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20261116 21:09:16.502501 1753 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.502671 1739 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20261116 21:09:16.502748 1739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal
uuid: "180facb734604feeabf76f00299f0451"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.502831 1739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:16.530956 1739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:16.531298 1739 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:16.531416 1739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:16.535614 1739 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:35923
I20261116 21:09:16.535643 1805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:35923 every 8 connection(s)
I20261116 21:09:16.536006 1739 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/data/info.pb
I20261116 21:09:16.536533 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1739
I20261116 21:09:16.536618 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/master-0/wal/instance
I20261116 21:09:16.536763 1806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:16.539664 1806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451: Bootstrap starting.
I20261116 21:09:16.540282 1806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:16.540558 1806 log.cc:826] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:16.541148 1806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451: No bootstrap required, opened a new log
I20261116 21:09:16.542681 1806 raft_consensus.cc:360] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } }
I20261116 21:09:16.542856 1806 raft_consensus.cc:386] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:16.542897 1806 raft_consensus.cc:748] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180facb734604feeabf76f00299f0451, State: Initialized, Role: FOLLOWER
I20261116 21:09:16.543004 1806 consensus_queue.cc:261] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } }
I20261116 21:09:16.543098 1806 raft_consensus.cc:400] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:16.543141 1806 raft_consensus.cc:499] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:16.543185 1806 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:16.543702 1806 raft_consensus.cc:521] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } }
I20261116 21:09:16.543850 1806 leader_election.cc:304] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 180facb734604feeabf76f00299f0451; no voters:
I20261116 21:09:16.544047 1806 leader_election.cc:290] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:16.544129 1811 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:16.544298 1806 sys_catalog.cc:565] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:16.544310 1811 raft_consensus.cc:705] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [term 1 LEADER]: Becoming Leader. State: Replica: 180facb734604feeabf76f00299f0451, State: Running, Role: LEADER
I20261116 21:09:16.544523 1811 consensus_queue.cc:238] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } }
I20261116 21:09:16.544968 1813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180facb734604feeabf76f00299f0451. Latest consensus state: current_term: 1 leader_uuid: "180facb734604feeabf76f00299f0451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } } }
I20261116 21:09:16.544945 1812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "180facb734604feeabf76f00299f0451" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180facb734604feeabf76f00299f0451" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 35923 } } }
I20261116 21:09:16.545099 1813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:16.545106 1812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:16.545498 1820 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:16.545966 1820 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:16.547384 1820 catalog_manager.cc:1398] Generated new cluster ID: a24da184339a448da18a61ecb29f8c68
I20261116 21:09:16.547431 1820 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:16.556442 1820 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:16.556980 1820 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:16.564543 1820 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 180facb734604feeabf76f00299f0451: Generated new TSK 0
I20261116 21:09:16.564738 1820 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:16.567488 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--builtin_ntp_servers=127.0.117.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:16.645068 1830 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261116 21:09:16.645279 1830 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:16.645308 1830 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:16.645382 1830 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:16.647073 1830 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:16.647171 1830 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:16.648764 1830 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--never_fsync=true
--heap_profile_path=/tmp/kudu.1830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:16.648996 1830 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:16.649187 1830 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:16.651602 1835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.651587 1836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.651786 1838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:16.652050 1830 server_base.cc:1061] running on GCE node
I20261116 21:09:16.652222 1830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:16.652439 1830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:16.653612 1830 hybrid_clock.cc:648] HybridClock initialized: now 1794863356653601 us; error 25 us; skew 500 ppm
I20261116 21:09:16.654688 1830 webserver.cc:577] Webserver started at http://127.0.117.193:44741/ using document root <none> and password file <none>
I20261116 21:09:16.654899 1830 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:16.654938 1830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:16.655043 1830 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:16.655895 1830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data/instance:
uuid: "355bc0ec70e84346834ed37aa981f498"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.656234 1830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal/instance:
uuid: "355bc0ec70e84346834ed37aa981f498"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.657445 1830 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:16.658342 1844 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.658522 1830 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:16.658576 1830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal
uuid: "355bc0ec70e84346834ed37aa981f498"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.658675 1830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:16.671995 1830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:16.672284 1830 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:16.672417 1830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:16.672658 1830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:16.673035 1830 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:16.673069 1830 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.673092 1830 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:16.673112 1830 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.680253 1830 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:39375
I20261116 21:09:16.680325 1957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:39375 every 8 connection(s)
I20261116 21:09:16.680651 1830 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/data/info.pb
I20261116 21:09:16.682317 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1830
I20261116 21:09:16.682418 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-0/wal/instance
I20261116 21:09:16.683687 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--builtin_ntp_servers=127.0.117.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261116 21:09:16.693401 1958 heartbeater.cc:345] Connected to a master server at 127.0.117.254:35923
I20261116 21:09:16.693523 1958 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:16.693753 1958 heartbeater.cc:508] Master 127.0.117.254:35923 requested a full tablet report, sending...
I20261116 21:09:16.694402 1770 ts_manager.cc:210] Registered new tserver with Master: 355bc0ec70e84346834ed37aa981f498 (127.0.117.193:39375)
I20261116 21:09:16.695418 1770 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:53049
W20261116 21:09:16.763404 1961 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261116 21:09:16.763592 1961 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:16.763618 1961 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:16.763646 1961 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:16.765177 1961 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:16.765256 1961 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:16.766844 1961 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--never_fsync=true
--heap_profile_path=/tmp/kudu.1961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:16.767072 1961 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:16.767261 1961 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:16.769676 1967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.769726 1969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.769673 1966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:16.769873 1961 server_base.cc:1061] running on GCE node
I20261116 21:09:16.770025 1961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:16.770231 1961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:16.771384 1961 hybrid_clock.cc:648] HybridClock initialized: now 1794863356771368 us; error 27 us; skew 500 ppm
I20261116 21:09:16.772310 1961 webserver.cc:577] Webserver started at http://127.0.117.194:38593/ using document root <none> and password file <none>
I20261116 21:09:16.772513 1961 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:16.772571 1961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:16.772677 1961 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:16.773597 1961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data/instance:
uuid: "43f73d14621a49c89a89f25c919aa8de"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.773933 1961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal/instance:
uuid: "43f73d14621a49c89a89f25c919aa8de"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.775053 1961 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:16.775738 1975 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.775952 1961 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:16.776028 1961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal
uuid: "43f73d14621a49c89a89f25c919aa8de"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.776085 1961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:16.787716 1961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:16.787998 1961 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:16.788146 1961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:16.788357 1961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:16.788690 1961 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:16.788745 1961 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.788787 1961 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:16.788815 1961 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.794868 1961 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:43237
I20261116 21:09:16.794950 2088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:43237 every 8 connection(s)
I20261116 21:09:16.795287 1961 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/data/info.pb
I20261116 21:09:16.798660 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 1961
I20261116 21:09:16.798785 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-1/wal/instance
I20261116 21:09:16.799561 2089 heartbeater.cc:345] Connected to a master server at 127.0.117.254:35923
I20261116 21:09:16.799672 2089 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:16.799870 2089 heartbeater.cc:508] Master 127.0.117.254:35923 requested a full tablet report, sending...
I20261116 21:09:16.800274 1769 ts_manager.cc:210] Registered new tserver with Master: 43f73d14621a49c89a89f25c919aa8de (127.0.117.194:43237)
I20261116 21:09:16.800480 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--builtin_ntp_servers=127.0.117.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
I20261116 21:09:16.800789 1769 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:51651
W20261116 21:09:16.878341 2092 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261116 21:09:16.878482 2092 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:16.878497 2092 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:16.878515 2092 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:16.880084 2092 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:16.880147 2092 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:16.881676 2092 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--never_fsync=true
--heap_profile_path=/tmp/kudu.2092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:16.881832 2092 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:16.881971 2092 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:16.884380 2098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.884390 2097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:16.884423 2100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:16.884720 2092 server_base.cc:1061] running on GCE node
I20261116 21:09:16.884857 2092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:16.885028 2092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:16.886175 2092 hybrid_clock.cc:648] HybridClock initialized: now 1794863356886153 us; error 34 us; skew 500 ppm
I20261116 21:09:16.887184 2092 webserver.cc:577] Webserver started at http://127.0.117.195:38345/ using document root <none> and password file <none>
I20261116 21:09:16.887342 2092 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:16.887377 2092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:16.887444 2092 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:16.888249 2092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data/instance:
uuid: "91eb66a6d3724b7aa8bb67550b6e7dad"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.888515 2092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal/instance:
uuid: "91eb66a6d3724b7aa8bb67550b6e7dad"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.889617 2092 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20261116 21:09:16.890385 2106 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.890635 2092 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:16.890707 2092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal
uuid: "91eb66a6d3724b7aa8bb67550b6e7dad"
format_stamp: "Formatted at 2026-11-16 21:09:16 on dist-test-slave-gg0q"
I20261116 21:09:16.890767 2092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:16.918277 2092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:16.918547 2092 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:16.918651 2092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:16.918836 2092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:16.919274 2092 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:16.919312 2092 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.919339 2092 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:16.919400 2092 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:16.925796 2092 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:39077
I20261116 21:09:16.925832 2219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:39077 every 8 connection(s)
I20261116 21:09:16.926206 2092 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/data/info.pb
I20261116 21:09:16.931236 2220 heartbeater.cc:345] Connected to a master server at 127.0.117.254:35923
I20261116 21:09:16.931354 2220 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:16.931550 2220 heartbeater.cc:508] Master 127.0.117.254:35923 requested a full tablet report, sending...
I20261116 21:09:16.931999 1769 ts_manager.cc:210] Registered new tserver with Master: 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195:39077)
I20261116 21:09:16.932466 1769 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:46581
I20261116 21:09:16.935513 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2092
I20261116 21:09:16.935599 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-2/wal/instance
I20261116 21:09:16.936685 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.196:0
--local_ip_for_outbound_sockets=127.0.117.196
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--builtin_ntp_servers=127.0.117.212:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--raft_prepare_replacement_before_eviction=true with env {}
W20261116 21:09:17.015080 2223 flags.cc:432] Enabled unsafe flag: --enable_leader_failure_detection=false
W20261116 21:09:17.015233 2223 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.015249 2223 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.015268 2223 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.016820 2223 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.016877 2223 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.196
I20261116 21:09:17.018491 2223 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:35923
--never_fsync=true
--heap_profile_path=/tmp/kudu.2223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.196
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.018694 2223 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.018874 2223 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.021430 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.021449 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.021592 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.021611 2223 server_base.cc:1061] running on GCE node
I20261116 21:09:17.021826 2223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.022022 2223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.023200 2223 hybrid_clock.cc:648] HybridClock initialized: now 1794863357023184 us; error 23 us; skew 500 ppm
I20261116 21:09:17.024272 2223 webserver.cc:577] Webserver started at http://127.0.117.196:45391/ using document root <none> and password file <none>
I20261116 21:09:17.024483 2223 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.024544 2223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.024649 2223 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.025556 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data/instance:
uuid: "fe0b133de2fa419aa1499ac54737bd74"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.025879 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal/instance:
uuid: "fe0b133de2fa419aa1499ac54737bd74"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.027029 2223 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:17.027762 2237 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.027931 2223 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:17.028007 2223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal
uuid: "fe0b133de2fa419aa1499ac54737bd74"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.028080 2223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.035867 2223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.036142 2223 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.036281 2223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.036474 2223 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:17.036782 2223 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:17.036834 2223 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.036875 2223 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:17.036901 2223 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.042866 2223 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.196:43397
I20261116 21:09:17.042940 2350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.196:43397 every 8 connection(s)
I20261116 21:09:17.043273 2223 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/data/info.pb
I20261116 21:09:17.048120 2351 heartbeater.cc:345] Connected to a master server at 127.0.117.254:35923
I20261116 21:09:17.048218 2351 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:17.048471 2351 heartbeater.cc:508] Master 127.0.117.254:35923 requested a full tablet report, sending...
I20261116 21:09:17.048950 1769 ts_manager.cc:210] Registered new tserver with Master: fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196:43397)
I20261116 21:09:17.049389 1769 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.196:42819
I20261116 21:09:17.050822 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2223
I20261116 21:09:17.050927 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.RaftConfigChangeITest.TestBulkChangeConfig.1794863344313111-471-0/minicluster-data/ts-3/wal/instance
I20261116 21:09:17.052076 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:17.064327 1769 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38390:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20261116 21:09:17.071472 2154 tablet_service.cc:1500] Processing CreateTablet for tablet cb55d0e6e23a4512b187e5bfcae9ee4c (DEFAULT_TABLE table=test-workload [id=abe3054abb094a32b4024a219b695103]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.071640 1892 tablet_service.cc:1500] Processing CreateTablet for tablet cb55d0e6e23a4512b187e5bfcae9ee4c (DEFAULT_TABLE table=test-workload [id=abe3054abb094a32b4024a219b695103]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.071784 2154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55d0e6e23a4512b187e5bfcae9ee4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.071973 1892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55d0e6e23a4512b187e5bfcae9ee4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.073998 2373 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: Bootstrap starting.
I20261116 21:09:17.074081 2374 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: Bootstrap starting.
I20261116 21:09:17.074621 2374 tablet_bootstrap.cc:654] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.074750 2373 tablet_bootstrap.cc:654] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.074906 2374 log.cc:826] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.075320 2373 log.cc:826] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.075491 2023 tablet_service.cc:1500] Processing CreateTablet for tablet cb55d0e6e23a4512b187e5bfcae9ee4c (DEFAULT_TABLE table=test-workload [id=abe3054abb094a32b4024a219b695103]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.075605 2374 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: No bootstrap required, opened a new log
I20261116 21:09:17.075685 2374 ts_tablet_manager.cc:1449] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.075783 2023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55d0e6e23a4512b187e5bfcae9ee4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.075942 2373 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: No bootstrap required, opened a new log
I20261116 21:09:17.076009 2373 ts_tablet_manager.cc:1449] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.077263 2374 raft_consensus.cc:360] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.077430 2374 raft_consensus.cc:748] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355bc0ec70e84346834ed37aa981f498, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.077502 2374 consensus_queue.cc:261] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.077567 2373 raft_consensus.cc:360] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.077698 2373 raft_consensus.cc:748] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91eb66a6d3724b7aa8bb67550b6e7dad, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.077754 2374 ts_tablet_manager.cc:1480] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:17.077757 2373 consensus_queue.cc:261] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.077864 1958 heartbeater.cc:500] Master 127.0.117.254:35923 was elected leader, sending a full tablet report...
I20261116 21:09:17.077970 2373 ts_tablet_manager.cc:1480] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:17.078069 2220 heartbeater.cc:500] Master 127.0.117.254:35923 was elected leader, sending a full tablet report...
I20261116 21:09:17.078078 2378 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: Bootstrap starting.
I20261116 21:09:17.079136 2378 tablet_bootstrap.cc:654] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.079430 2378 log.cc:826] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.080009 2378 tablet_bootstrap.cc:492] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: No bootstrap required, opened a new log
I20261116 21:09:17.080092 2378 ts_tablet_manager.cc:1449] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.081284 2378 raft_consensus.cc:360] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.081483 2378 raft_consensus.cc:748] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f73d14621a49c89a89f25c919aa8de, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.081554 2378 consensus_queue.cc:261] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.081781 2378 ts_tablet_manager.cc:1480] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:17.081843 2089 heartbeater.cc:500] Master 127.0.117.254:35923 was elected leader, sending a full tablet report...
I20261116 21:09:17.094839 2043 tablet_service.cc:2033] Received Run Leader Election RPC: tablet_id: "cb55d0e6e23a4512b187e5bfcae9ee4c"
dest_uuid: "43f73d14621a49c89a89f25c919aa8de"
from {username='slave'} at 127.0.0.1:45596
I20261116 21:09:17.094949 2043 raft_consensus.cc:499] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20261116 21:09:17.094987 2043 raft_consensus.cc:3096] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.095693 2043 raft_consensus.cc:521] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.095999 2043 leader_election.cc:290] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [CANDIDATE]: Term 1 election: Requested vote from peers 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195:39077), 355bc0ec70e84346834ed37aa981f498 (127.0.117.193:39375)
I20261116 21:09:17.100008 2174 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cb55d0e6e23a4512b187e5bfcae9ee4c" candidate_uuid: "43f73d14621a49c89a89f25c919aa8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad"
I20261116 21:09:17.100018 1912 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "cb55d0e6e23a4512b187e5bfcae9ee4c" candidate_uuid: "43f73d14621a49c89a89f25c919aa8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "355bc0ec70e84346834ed37aa981f498"
I20261116 21:09:17.100142 1912 raft_consensus.cc:3096] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.100142 2174 raft_consensus.cc:3096] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.100960 2174 raft_consensus.cc:2490] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f73d14621a49c89a89f25c919aa8de in term 1.
I20261116 21:09:17.100960 1912 raft_consensus.cc:2490] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43f73d14621a49c89a89f25c919aa8de in term 1.
I20261116 21:09:17.101199 1979 leader_election.cc:304] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f73d14621a49c89a89f25c919aa8de, 91eb66a6d3724b7aa8bb67550b6e7dad; no voters:
I20261116 21:09:17.101459 2381 raft_consensus.cc:2826] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:17.101626 2381 raft_consensus.cc:705] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Becoming Leader. State: Replica: 43f73d14621a49c89a89f25c919aa8de, State: Running, Role: LEADER
I20261116 21:09:17.101735 2381 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.103433 2381 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:17.103641 2390 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:17.103847 1770 catalog_manager.cc:5840] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de reported cstate change: term changed from 0 to 1, leader changed from <none> to 43f73d14621a49c89a89f25c919aa8de (127.0.117.194). New cstate: current_term: 1 leader_uuid: "43f73d14621a49c89a89f25c919aa8de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } health_report { overall_health: HEALTHY } } }
W20261116 21:09:17.109294 2133 tablet_service.cc:720] failed op from {username='slave'} at 127.0.0.1:54628: Illegal state: replica 91eb66a6d3724b7aa8bb67550b6e7dad is not leader of this config: current role FOLLOWER
W20261116 21:09:17.109288 2134 tablet_service.cc:720] failed op from {username='slave'} at 127.0.0.1:54628: Illegal state: replica 91eb66a6d3724b7aa8bb67550b6e7dad is not leader of this config: current role FOLLOWER
I20261116 21:09:17.118371 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1739
I20261116 21:09:17.127051 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.127113 471 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.129777 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.130527 2174 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261116 21:09:17.130849 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20261116 21:09:17.130821 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20261116 21:09:17.131031 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20261116 21:09:17.135120 1978 consensus_peers.cc:597] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de -> Peer fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196:43397): Couldn't send request to peer fe0b133de2fa419aa1499ac54737bd74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb55d0e6e23a4512b187e5bfcae9ee4c. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:17.137799 2391 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
I20261116 21:09:17.139369 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
I20261116 21:09:17.139586 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index -1 to 10, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
W20261116 21:09:17.140221 2089 heartbeater.cc:648] Failed to heartbeat to 127.0.117.254:35923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.117.254:35923: connect: Connection refused (error 111)
I20261116 21:09:17.140792 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.140847 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.140910 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } }
W20261116 21:09:17.141028 1958 heartbeater.cc:648] Failed to heartbeat to 127.0.117.254:35923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.117.254:35923: connect: Connection refused (error 111)
W20261116 21:09:17.141191 2220 heartbeater.cc:648] Failed to heartbeat to 127.0.117.254:35923 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.117.254:35923: connect: Connection refused (error 111)
I20261116 21:09:17.141390 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } }
I20261116 21:09:17.141861 2174 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20261116 21:09:17.141896 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20261116 21:09:17.142023 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20261116 21:09:17.142088 2381 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20261116 21:09:17.142230 1978 consensus_peers.cc:597] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de -> Peer fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196:43397): Couldn't send request to peer fe0b133de2fa419aa1499ac54737bd74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb55d0e6e23a4512b187e5bfcae9ee4c. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:17.142609 2381 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.142730 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.143874 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) changed from VOTER to NON_VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.144227 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.144284 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } }
I20261116 21:09:17.144548 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } }
I20261116 21:09:17.145130 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20261116 21:09:17.145186 2174 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 LEARNER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20261116 21:09:17.145385 2409 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20261116 21:09:17.145478 2391 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20261116 21:09:17.145668 1978 consensus_peers.cc:597] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de -> Peer fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196:43397): Couldn't send request to peer fe0b133de2fa419aa1499ac54737bd74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb55d0e6e23a4512b187e5bfcae9ee4c. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:17.146178 2391 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.146281 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.147560 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.147547 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } } }
I20261116 21:09:17.147634 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.147694 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.148128 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.148192 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: NON_VOTER last_known_addr { host: "127.0.117.194" port: 43237 } attrs { promote: false replace: false } }
I20261116 21:09:17.148931 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.148990 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: true replace: false } }
I20261116 21:09:17.149582 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.149617 471 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.149643 471 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.149927 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }
I20261116 21:09:17.150288 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20261116 21:09:17.150480 2381 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20261116 21:09:17.151135 2390 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) evicted, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } }
I20261116 21:09:17.151325 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) evicted, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } }
I20261116 21:09:17.152680 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.152752 471 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.152805 471 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.153307 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.153528 471 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.153779 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.154213 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20261116 21:09:17.154304 2174 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 LEARNER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20261116 21:09:17.154508 2381 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20261116 21:09:17.154598 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20261116 21:09:17.155138 2174 raft_consensus.cc:879] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
W20261116 21:09:17.155194 2174 raft_consensus.cc:1590] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 14' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: NON_VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } } peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: true replace: false } }.
Pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } }:
New config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.155272 2381 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } }
I20261116 21:09:17.155308 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) evicted, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } }
I20261116 21:09:17.156055 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } }
I20261116 21:09:17.156792 2174 raft_consensus.cc:1491] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20261116 21:09:17.156904 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER 91eb66a6d3724b7aa8bb67550b6e7dad (127.0.117.195) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } }
I20261116 21:09:17.157035 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.157071 471 raft_config_change-itest.cc:351] type: ADD_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.157284 2043 consensus_queue.cc:238] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.157897 2173 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
W20261116 21:09:17.157967 1978 consensus_peers.cc:597] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de -> Peer fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196:43397): Couldn't send request to peer fe0b133de2fa419aa1499ac54737bd74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb55d0e6e23a4512b187e5bfcae9ee4c. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:17.158005 1912 raft_consensus.cc:1285] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Refusing update from remote peer 43f73d14621a49c89a89f25c919aa8de: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20261116 21:09:17.158109 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20261116 21:09:17.158272 2388 consensus_queue.cc:1049] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20261116 21:09:17.158917 2390 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 43f73d14621a49c89a89f25c919aa8de [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
I20261116 21:09:17.159067 2174 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 91eb66a6d3724b7aa8bb67550b6e7dad [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
I20261116 21:09:17.159860 1912 raft_consensus.cc:2983] T cb55d0e6e23a4512b187e5bfcae9ee4c P 355bc0ec70e84346834ed37aa981f498 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER fe0b133de2fa419aa1499ac54737bd74 (127.0.117.196) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "355bc0ec70e84346834ed37aa981f498" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39375 } } peers { permanent_uuid: "43f73d14621a49c89a89f25c919aa8de" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 43237 } } peers { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } } peers { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } } }
I20261116 21:09:17.160379 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.160425 471 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.160482 471 raft_config_change-itest.cc:351] type: REMOVE_PEER peer { permanent_uuid: "fe0b133de2fa419aa1499ac54737bd74" member_type: UNKNOWN_MEMBER_TYPE last_known_addr { host: "127.0.117.196" port: 43397 } attrs { promote: false replace: false } }
I20261116 21:09:17.161049 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.161110 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: false } }
I20261116 21:09:17.161559 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.162000 471 raft_config_change-itest.cc:349] submitting config change with changes:
I20261116 21:09:17.162053 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: false replace: true } }
I20261116 21:09:17.162103 471 raft_config_change-itest.cc:351] type: MODIFY_PEER peer { permanent_uuid: "91eb66a6d3724b7aa8bb67550b6e7dad" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39077 } attrs { promote: true replace: false } }
I20261116 21:09:17.163057 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1830
I20261116 21:09:17.170454 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 1961
I20261116 21:09:17.179277 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2092
I20261116 21:09:17.186643 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2223
2026-11-16T21:09:17Z chronyd exiting
[ OK ] RaftConfigChangeITest.TestBulkChangeConfig (803 ms)
[----------] 3 tests from RaftConfigChangeITest (12890 ms total)
[----------] 3 tests from SlowTabletBootstrapTest
[ RUN ] SlowTabletBootstrapTest.TestSlowBootstrap
I20261116 21:09:17.209685 471 ts_itest-base.cc:115] Starting cluster with:
I20261116 21:09:17.209748 471 ts_itest-base.cc:116] --------------
I20261116 21:09:17.209769 471 ts_itest-base.cc:117] 4 tablet servers
I20261116 21:09:17.209796 471 ts_itest-base.cc:118] 3 replicas per TS
I20261116 21:09:17.209822 471 ts_itest-base.cc:119] --------------
2026-11-16T21:09:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:17Z Disabled control of system clock
I20261116 21:09:17.217252 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:43825
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:43825 with env {}
W20261116 21:09:17.294073 2417 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.294267 2417 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.294299 2417 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.295838 2417 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:17.295919 2417 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.295941 2417 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:17.295960 2417 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:17.297644 2417 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:43825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.297899 2417 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.298099 2417 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.300352 2422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.300374 2423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.300419 2425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.300419 2417 server_base.cc:1061] running on GCE node
I20261116 21:09:17.300769 2417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.301013 2417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.302151 2417 hybrid_clock.cc:648] HybridClock initialized: now 1794863357302131 us; error 42 us; skew 500 ppm
I20261116 21:09:17.303203 2417 webserver.cc:577] Webserver started at http://127.0.117.254:34575/ using document root <none> and password file <none>
I20261116 21:09:17.303431 2417 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.303475 2417 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.303579 2417 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.304507 2417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6de23e57ce4046a9812097b9f7b44368"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.304886 2417 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6de23e57ce4046a9812097b9f7b44368"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.306295 2417 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20261116 21:09:17.307068 2431 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.307256 2417 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:17.307315 2417 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6de23e57ce4046a9812097b9f7b44368"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.307384 2417 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.322137 2417 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.322440 2417 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.322576 2417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.326817 2417 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:43825
I20261116 21:09:17.326850 2483 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:43825 every 8 connection(s)
I20261116 21:09:17.327222 2417 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261116 21:09:17.327841 2484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.330322 2484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368: Bootstrap starting.
I20261116 21:09:17.330936 2484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.331208 2484 log.cc:826] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.331451 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2417
I20261116 21:09:17.331547 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261116 21:09:17.331959 2484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368: No bootstrap required, opened a new log
I20261116 21:09:17.333953 2484 raft_consensus.cc:360] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } }
I20261116 21:09:17.334108 2484 raft_consensus.cc:386] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:17.334136 2484 raft_consensus.cc:748] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6de23e57ce4046a9812097b9f7b44368, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.334223 2484 consensus_queue.cc:261] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } }
I20261116 21:09:17.334282 2484 raft_consensus.cc:400] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:17.334316 2484 raft_consensus.cc:499] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:17.334358 2484 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.335021 2484 raft_consensus.cc:521] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } }
I20261116 21:09:17.335148 2484 leader_election.cc:304] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6de23e57ce4046a9812097b9f7b44368; no voters:
I20261116 21:09:17.335346 2484 leader_election.cc:290] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:17.335407 2489 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:17.335546 2489 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [term 1 LEADER]: Becoming Leader. State: Replica: 6de23e57ce4046a9812097b9f7b44368, State: Running, Role: LEADER
I20261116 21:09:17.335598 2484 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:17.335633 2489 consensus_queue.cc:238] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } }
I20261116 21:09:17.335996 2489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6de23e57ce4046a9812097b9f7b44368. Latest consensus state: current_term: 1 leader_uuid: "6de23e57ce4046a9812097b9f7b44368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } } }
I20261116 21:09:17.336364 2491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6de23e57ce4046a9812097b9f7b44368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6de23e57ce4046a9812097b9f7b44368" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 43825 } } }
I20261116 21:09:17.336072 2489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:17.336676 2491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:17.336822 2498 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:17.337399 2498 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:17.338804 2498 catalog_manager.cc:1398] Generated new cluster ID: 52bdd503e78048eba3f54aea439eae58
I20261116 21:09:17.338855 2498 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:17.356032 2498 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:17.356843 2498 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:17.368750 2498 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 6de23e57ce4046a9812097b9f7b44368: Generated new TSK 0
I20261116 21:09:17.368978 2498 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:17.371958 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:17.449132 2508 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:17.449334 2508 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.449398 2508 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.449431 2508 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.450947 2508 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:17.451016 2508 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:17.451047 2508 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.451089 2508 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:17.451138 2508 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:17.452723 2508 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--never_fsync=true
--heap_profile_path=/tmp/kudu.2508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.452965 2508 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.453148 2508 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.455775 2513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.455826 2516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.455786 2514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.455799 2508 server_base.cc:1061] running on GCE node
I20261116 21:09:17.456152 2508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.456415 2508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.457554 2508 hybrid_clock.cc:648] HybridClock initialized: now 1794863357457543 us; error 34 us; skew 500 ppm
I20261116 21:09:17.458681 2508 webserver.cc:577] Webserver started at http://127.0.117.193:33503/ using document root <none> and password file <none>
I20261116 21:09:17.458932 2508 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.458973 2508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.459082 2508 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.459941 2508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9ec683f2b7b349feb4369d58cbddb135"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.460278 2508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9ec683f2b7b349feb4369d58cbddb135"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.461508 2508 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20261116 21:09:17.462303 2522 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.462476 2508 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:17.462533 2508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ec683f2b7b349feb4369d58cbddb135"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.462611 2508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.472471 2508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.472683 2508 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.472800 2508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.473001 2508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:17.473304 2508 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:17.473383 2508 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.473426 2508 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:17.473450 2508 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.480294 2508 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:38435
I20261116 21:09:17.480369 2635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:38435 every 8 connection(s)
I20261116 21:09:17.480674 2508 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261116 21:09:17.484997 2636 heartbeater.cc:345] Connected to a master server at 127.0.117.254:43825
I20261116 21:09:17.485097 2636 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:17.485287 2636 heartbeater.cc:508] Master 127.0.117.254:43825 requested a full tablet report, sending...
I20261116 21:09:17.485864 2447 ts_manager.cc:210] Registered new tserver with Master: 9ec683f2b7b349feb4369d58cbddb135 (127.0.117.193:38435)
I20261116 21:09:17.486529 2447 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:40775
I20261116 21:09:17.486838 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2508
I20261116 21:09:17.486934 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261116 21:09:17.487988 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:17.566288 2639 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:17.566439 2639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.566457 2639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.566473 2639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.568013 2639 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:17.568065 2639 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:17.568082 2639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.568104 2639 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:17.568118 2639 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:17.569723 2639 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--never_fsync=true
--heap_profile_path=/tmp/kudu.2639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.569919 2639 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.570184 2639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.572736 2645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.572716 2644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.572861 2647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.573475 2639 server_base.cc:1061] running on GCE node
I20261116 21:09:17.573704 2639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.573932 2639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.575084 2639 hybrid_clock.cc:648] HybridClock initialized: now 1794863357575074 us; error 29 us; skew 500 ppm
I20261116 21:09:17.576227 2639 webserver.cc:577] Webserver started at http://127.0.117.194:43445/ using document root <none> and password file <none>
I20261116 21:09:17.576447 2639 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.576488 2639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.576588 2639 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.577512 2639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "58a8b52f54d74293b72dc7e8292e7bb6"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.577872 2639 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "58a8b52f54d74293b72dc7e8292e7bb6"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.578989 2639 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:17.579720 2653 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.579881 2639 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:17.579957 2639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "58a8b52f54d74293b72dc7e8292e7bb6"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.580029 2639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.592067 2639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.592320 2639 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.592453 2639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.592659 2639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:17.592998 2639 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:17.593052 2639 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.593093 2639 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:17.593122 2639 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.599030 2639 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:41035
I20261116 21:09:17.599071 2766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:41035 every 8 connection(s)
I20261116 21:09:17.599381 2639 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261116 21:09:17.602545 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2639
I20261116 21:09:17.602684 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261116 21:09:17.604094 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261116 21:09:17.604352 2767 heartbeater.cc:345] Connected to a master server at 127.0.117.254:43825
I20261116 21:09:17.604470 2767 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:17.604661 2767 heartbeater.cc:508] Master 127.0.117.254:43825 requested a full tablet report, sending...
I20261116 21:09:17.605078 2447 ts_manager.cc:210] Registered new tserver with Master: 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035)
I20261116 21:09:17.605682 2447 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:38899
W20261116 21:09:17.681734 2770 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:17.681878 2770 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.681893 2770 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.681909 2770 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.683532 2770 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:17.683578 2770 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:17.683593 2770 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.683614 2770 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:17.683627 2770 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:17.685191 2770 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.685405 2770 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.685567 2770 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.687985 2778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.688001 2776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.688001 2775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.688233 2770 server_base.cc:1061] running on GCE node
I20261116 21:09:17.688422 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.688632 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.689776 2770 hybrid_clock.cc:648] HybridClock initialized: now 1794863357689767 us; error 28 us; skew 500 ppm
I20261116 21:09:17.690927 2770 webserver.cc:577] Webserver started at http://127.0.117.195:33901/ using document root <none> and password file <none>
I20261116 21:09:17.691161 2770 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.691202 2770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.691305 2770 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.692124 2770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6e18af5b2a644fa0ac626f01a4cecb20"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.692448 2770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6e18af5b2a644fa0ac626f01a4cecb20"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.693703 2770 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20261116 21:09:17.694518 2784 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.694711 2770 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:17.694789 2770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6e18af5b2a644fa0ac626f01a4cecb20"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.694867 2770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.710299 2770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.710539 2770 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.710669 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.710862 2770 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:17.711274 2770 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:17.711316 2770 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.711355 2770 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:17.711385 2770 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.716897 2770 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:43671
I20261116 21:09:17.716938 2897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:43671 every 8 connection(s)
I20261116 21:09:17.717258 2770 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261116 21:09:17.719105 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2770
I20261116 21:09:17.719216 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261116 21:09:17.720492 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.196:0
--local_ip_for_outbound_sockets=127.0.117.196
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261116 21:09:17.722268 2898 heartbeater.cc:345] Connected to a master server at 127.0.117.254:43825
I20261116 21:09:17.722397 2898 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:17.722596 2898 heartbeater.cc:508] Master 127.0.117.254:43825 requested a full tablet report, sending...
I20261116 21:09:17.723026 2447 ts_manager.cc:210] Registered new tserver with Master: 6e18af5b2a644fa0ac626f01a4cecb20 (127.0.117.195:43671)
I20261116 21:09:17.723567 2447 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:35899
W20261116 21:09:17.799896 2901 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:17.800139 2901 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:17.800158 2901 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:17.800176 2901 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:17.801677 2901 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:17.801716 2901 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:17.801733 2901 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:17.801754 2901 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:17.801767 2901 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.196
I20261116 21:09:17.803267 2901 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:43825
--never_fsync=true
--heap_profile_path=/tmp/kudu.2901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.196
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:17.803418 2901 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:17.803548 2901 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:17.805915 2906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.805948 2907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:17.805949 2909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:17.806031 2901 server_base.cc:1061] running on GCE node
I20261116 21:09:17.806289 2901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:17.806437 2901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:17.807554 2901 hybrid_clock.cc:648] HybridClock initialized: now 1794863357807543 us; error 27 us; skew 500 ppm
I20261116 21:09:17.808552 2901 webserver.cc:577] Webserver started at http://127.0.117.196:36951/ using document root <none> and password file <none>
I20261116 21:09:17.808758 2901 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:17.808797 2901 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:17.808892 2901 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:17.809873 2901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b964a884f72b402982b46c1870c588be"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.810210 2901 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b964a884f72b402982b46c1870c588be"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.811414 2901 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:17.812196 2915 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.812376 2901 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:17.812460 2901 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b964a884f72b402982b46c1870c588be"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:17.812516 2901 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:17.822997 2901 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:17.823235 2901 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:17.823359 2901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:17.823580 2901 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:17.823915 2901 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:17.823945 2901 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.823967 2901 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:17.824016 2901 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:17.829564 2901 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.196:38437
I20261116 21:09:17.829603 3028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.196:38437 every 8 connection(s)
I20261116 21:09:17.829909 2901 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261116 21:09:17.834270 3029 heartbeater.cc:345] Connected to a master server at 127.0.117.254:43825
I20261116 21:09:17.834393 3029 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:17.834585 3029 heartbeater.cc:508] Master 127.0.117.254:43825 requested a full tablet report, sending...
I20261116 21:09:17.834993 2447 ts_manager.cc:210] Registered new tserver with Master: b964a884f72b402982b46c1870c588be (127.0.117.196:38437)
I20261116 21:09:17.835408 2447 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.196:43005
I20261116 21:09:17.835875 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 2901
I20261116 21:09:17.836000 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261116 21:09:17.837085 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:17.843019 2447 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57176:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261116 21:09:17.851058 2832 tablet_service.cc:1500] Processing CreateTablet for tablet b5563c851df34a0fb67eabdf7614113b (DEFAULT_TABLE table=TestTable [id=b6018c5ad4a94f71aee90fa9e262d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.851374 2832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5563c851df34a0fb67eabdf7614113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.851462 2701 tablet_service.cc:1500] Processing CreateTablet for tablet b5563c851df34a0fb67eabdf7614113b (DEFAULT_TABLE table=TestTable [id=b6018c5ad4a94f71aee90fa9e262d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.851505 2963 tablet_service.cc:1500] Processing CreateTablet for tablet b5563c851df34a0fb67eabdf7614113b (DEFAULT_TABLE table=TestTable [id=b6018c5ad4a94f71aee90fa9e262d564]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:17.851716 2963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5563c851df34a0fb67eabdf7614113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.851771 2701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5563c851df34a0fb67eabdf7614113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:17.853816 3043 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Bootstrap starting.
I20261116 21:09:17.854223 3044 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: Bootstrap starting.
I20261116 21:09:17.854460 3043 tablet_bootstrap.cc:654] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.854723 3043 log.cc:826] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.854835 3044 tablet_bootstrap.cc:654] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.855098 3044 log.cc:826] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.855392 3043 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: No bootstrap required, opened a new log
I20261116 21:09:17.855383 3045 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: Bootstrap starting.
I20261116 21:09:17.855465 3043 ts_tablet_manager.cc:1449] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.855888 3045 tablet_bootstrap.cc:654] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:17.856143 3045 log.cc:826] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:17.856560 3044 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: No bootstrap required, opened a new log
I20261116 21:09:17.856628 3044 ts_tablet_manager.cc:1449] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.856791 3045 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: No bootstrap required, opened a new log
I20261116 21:09:17.856849 3045 ts_tablet_manager.cc:1449] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:17.856997 3043 raft_consensus.cc:360] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.857148 3043 raft_consensus.cc:386] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:17.857184 3043 raft_consensus.cc:748] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a8b52f54d74293b72dc7e8292e7bb6, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.857290 3043 consensus_queue.cc:261] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.857558 3043 ts_tablet_manager.cc:1480] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:17.857575 2767 heartbeater.cc:500] Master 127.0.117.254:43825 was elected leader, sending a full tablet report...
I20261116 21:09:17.858668 3045 raft_consensus.cc:360] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.858798 3045 raft_consensus.cc:386] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:17.858825 3045 raft_consensus.cc:748] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b964a884f72b402982b46c1870c588be, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.858791 3044 raft_consensus.cc:360] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.858903 3044 raft_consensus.cc:386] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:17.858937 3044 raft_consensus.cc:748] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e18af5b2a644fa0ac626f01a4cecb20, State: Initialized, Role: FOLLOWER
I20261116 21:09:17.858924 3045 consensus_queue.cc:261] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.859010 3044 consensus_queue.cc:261] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.859207 3029 heartbeater.cc:500] Master 127.0.117.254:43825 was elected leader, sending a full tablet report...
I20261116 21:09:17.859211 3044 ts_tablet_manager.cc:1480] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261116 21:09:17.859212 3045 ts_tablet_manager.cc:1480] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:17.859480 2898 heartbeater.cc:500] Master 127.0.117.254:43825 was elected leader, sending a full tablet report...
I20261116 21:09:17.885149 3050 raft_consensus.cc:499] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:17.885262 3050 raft_consensus.cc:521] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.885702 3050 leader_election.cc:290] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035), 6e18af5b2a644fa0ac626f01a4cecb20 (127.0.117.195:43671)
I20261116 21:09:17.888988 2721 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b5563c851df34a0fb67eabdf7614113b" candidate_uuid: "b964a884f72b402982b46c1870c588be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" is_pre_election: true
I20261116 21:09:17.889207 2721 raft_consensus.cc:2490] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b964a884f72b402982b46c1870c588be in term 0.
I20261116 21:09:17.889230 2852 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b5563c851df34a0fb67eabdf7614113b" candidate_uuid: "b964a884f72b402982b46c1870c588be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" is_pre_election: true
I20261116 21:09:17.889403 2852 raft_consensus.cc:2490] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b964a884f72b402982b46c1870c588be in term 0.
I20261116 21:09:17.889452 2919 leader_election.cc:304] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58a8b52f54d74293b72dc7e8292e7bb6, b964a884f72b402982b46c1870c588be; no voters:
I20261116 21:09:17.889581 3050 raft_consensus.cc:2826] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 21:09:17.889652 3050 raft_consensus.cc:499] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:17.889678 3050 raft_consensus.cc:3096] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.890331 3050 raft_consensus.cc:521] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.890450 3050 leader_election.cc:290] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [CANDIDATE]: Term 1 election: Requested vote from peers 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035), 6e18af5b2a644fa0ac626f01a4cecb20 (127.0.117.195:43671)
I20261116 21:09:17.890659 2852 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b5563c851df34a0fb67eabdf7614113b" candidate_uuid: "b964a884f72b402982b46c1870c588be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e18af5b2a644fa0ac626f01a4cecb20"
I20261116 21:09:17.890729 2852 raft_consensus.cc:3096] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.891335 2852 raft_consensus.cc:2490] T b5563c851df34a0fb67eabdf7614113b P 6e18af5b2a644fa0ac626f01a4cecb20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b964a884f72b402982b46c1870c588be in term 1.
I20261116 21:09:17.891465 2721 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "b5563c851df34a0fb67eabdf7614113b" candidate_uuid: "b964a884f72b402982b46c1870c588be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a8b52f54d74293b72dc7e8292e7bb6"
I20261116 21:09:17.891556 2721 raft_consensus.cc:3096] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:17.891543 2916 leader_election.cc:304] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e18af5b2a644fa0ac626f01a4cecb20, b964a884f72b402982b46c1870c588be; no voters:
I20261116 21:09:17.891671 3050 raft_consensus.cc:2826] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:17.891875 3050 raft_consensus.cc:705] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [term 1 LEADER]: Becoming Leader. State: Replica: b964a884f72b402982b46c1870c588be, State: Running, Role: LEADER
I20261116 21:09:17.892134 3050 consensus_queue.cc:238] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:17.892645 2721 raft_consensus.cc:2490] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b964a884f72b402982b46c1870c588be in term 1.
I20261116 21:09:17.893306 3055 consensus_queue.cc:1049] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:17.893286 2448 catalog_manager.cc:5840] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be reported cstate change: term changed from 0 to 1, leader changed from <none> to b964a884f72b402982b46c1870c588be (127.0.117.196). New cstate: current_term: 1 leader_uuid: "b964a884f72b402982b46c1870c588be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } health_report { overall_health: HEALTHY } } }
I20261116 21:09:17.893582 3053 consensus_queue.cc:1049] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:17.908789 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:17.909361 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 58a8b52f54d74293b72dc7e8292e7bb6 to finish bootstrapping
I20261116 21:09:17.913753 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6e18af5b2a644fa0ac626f01a4cecb20 to finish bootstrapping
I20261116 21:09:17.916934 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b964a884f72b402982b46c1870c588be to finish bootstrapping
I20261116 21:09:17.920331 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2639
I20261116 21:09:17.926303 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:41035
--local_ip_for_outbound_sockets=127.0.117.194
--tserver_master_addrs=127.0.117.254:43825
--webserver_port=43445
--webserver_interface=127.0.117.194
--builtin_ntp_servers=127.0.117.212:37057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=4000 with env {}
W20261116 21:09:18.006012 3062 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:18.006171 3062 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:18.006187 3062 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:18.006206 3062 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=4000
W20261116 21:09:18.006217 3062 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:18.007791 3062 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:18.007835 3062 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:18.007851 3062 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:18.007875 3062 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:18.007915 3062 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:18.009608 3062 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:37057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:41035
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=43445
--tserver_master_addrs=127.0.117.254:43825
--tablet_bootstrap_inject_latency_ms=4000
--never_fsync=true
--heap_profile_path=/tmp/kudu.3062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:18.009824 3062 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:18.010051 3062 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:18.012640 3070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:18.012651 3068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:18.012677 3067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:18.012679 3062 server_base.cc:1061] running on GCE node
I20261116 21:09:18.013026 3062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:18.013286 3062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:18.014485 3062 hybrid_clock.cc:648] HybridClock initialized: now 1794863358014473 us; error 34 us; skew 500 ppm
I20261116 21:09:18.015497 3062 webserver.cc:577] Webserver started at http://127.0.117.194:43445/ using document root <none> and password file <none>
I20261116 21:09:18.015730 3062 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:18.015807 3062 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:18.016981 3062 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:18.017704 3076 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:18.017870 3062 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:18.017947 3062 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "58a8b52f54d74293b72dc7e8292e7bb6"
format_stamp: "Formatted at 2026-11-16 21:09:17 on dist-test-slave-gg0q"
I20261116 21:09:18.018278 3062 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:18.029838 3062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:18.030069 3062 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:18.030210 3062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:18.030423 3062 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:18.030920 3083 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261116 21:09:18.031874 3062 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261116 21:09:18.031960 3062 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:18.032016 3062 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261116 21:09:18.032572 3062 ts_tablet_manager.cc:662] Registered 1 tablets
W20261116 21:09:18.032616 3083 ts_tablet_manager.cc:1428] Injecting 4000ms delay in tablet bootstrapping
I20261116 21:09:18.032630 3062 ts_tablet_manager.cc:641] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:18.038913 3062 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:41035
I20261116 21:09:18.038960 3190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:41035 every 8 connection(s)
I20261116 21:09:18.039311 3062 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261116 21:09:18.040891 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3062
I20261116 21:09:18.044238 3191 heartbeater.cc:345] Connected to a master server at 127.0.117.254:43825
I20261116 21:09:18.044353 3191 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:18.044546 3191 heartbeater.cc:508] Master 127.0.117.254:43825 requested a full tablet report, sending...
I20261116 21:09:18.045070 2446 ts_manager.cc:210] Re-registered known tserver with Master: 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035)
I20261116 21:09:18.045548 2446 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:41915
I20261116 21:09:18.487425 2636 heartbeater.cc:500] Master 127.0.117.254:43825 was elected leader, sending a full tablet report...
W20261116 21:09:18.513540 2919 consensus_peers.cc:597] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be -> Peer 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035): Couldn't send request to peer 58a8b52f54d74293b72dc7e8292e7bb6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:19.046325 3191 heartbeater.cc:500] Master 127.0.117.254:43825 was elected leader, sending a full tablet report...
W20261116 21:09:20.834532 2919 consensus_peers.cc:597] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be -> Peer 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035): Couldn't send request to peer 58a8b52f54d74293b72dc7e8292e7bb6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20261116 21:09:22.032987 3083 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Bootstrap starting.
I20261116 21:09:22.035917 3083 log.cc:826] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:22.036993 3083 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 21:09:22.037276 3083 tablet_bootstrap.cc:492] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Bootstrap complete.
I20261116 21:09:22.037533 3083 ts_tablet_manager.cc:1449] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20261116 21:09:22.038564 3083 raft_consensus.cc:360] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:22.038754 3083 raft_consensus.cc:748] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a8b52f54d74293b72dc7e8292e7bb6, State: Initialized, Role: FOLLOWER
I20261116 21:09:22.038868 3083 consensus_queue.cc:261] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58a8b52f54d74293b72dc7e8292e7bb6" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 41035 } } peers { permanent_uuid: "6e18af5b2a644fa0ac626f01a4cecb20" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 43671 } } peers { permanent_uuid: "b964a884f72b402982b46c1870c588be" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 38437 } }
I20261116 21:09:22.039129 3083 ts_tablet_manager.cc:1480] T b5563c851df34a0fb67eabdf7614113b P 58a8b52f54d74293b72dc7e8292e7bb6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:25.051689 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2508
I20261116 21:09:25.056777 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3062
W20261116 21:09:25.061857 2919 connection.cc:573] client connection to 127.0.117.194:41035 recv error: Network error: recv error from unknown peer: Connection reset by peer (error 104)
W20261116 21:09:25.061949 2919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Connection reset by peer (error 104)
I20261116 21:09:25.062161 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2770
W20261116 21:09:25.063555 2919 consensus_peers.cc:597] T b5563c851df34a0fb67eabdf7614113b P b964a884f72b402982b46c1870c588be -> Peer 58a8b52f54d74293b72dc7e8292e7bb6 (127.0.117.194:41035): Couldn't send request to peer 58a8b52f54d74293b72dc7e8292e7bb6. Status: Network error: Client connection negotiation failed: client connection to 127.0.117.194:41035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:25.067243 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2901
I20261116 21:09:25.072397 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 2417
2026-11-16T21:09:25Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestSlowBootstrap (7884 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehind
I20261116 21:09:25.094093 471 ts_itest-base.cc:115] Starting cluster with:
I20261116 21:09:25.094151 471 ts_itest-base.cc:116] --------------
I20261116 21:09:25.094163 471 ts_itest-base.cc:117] 4 tablet servers
I20261116 21:09:25.094174 471 ts_itest-base.cc:118] 3 replicas per TS
I20261116 21:09:25.094185 471 ts_itest-base.cc:119] --------------
2026-11-16T21:09:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:25Z Disabled control of system clock
I20261116 21:09:25.101975 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:36535
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:36535 with env {}
W20261116 21:09:25.181044 3215 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:25.181203 3215 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:25.181222 3215 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:25.182832 3215 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:25.182886 3215 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:25.182901 3215 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:25.182914 3215 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:25.184545 3215 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:36535
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:25.184764 3215 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:25.184947 3215 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:25.188154 3221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.188225 3220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.188274 3215 server_base.cc:1061] running on GCE node
W20261116 21:09:25.188347 3223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.188576 3215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:25.188836 3215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:25.189997 3215 hybrid_clock.cc:648] HybridClock initialized: now 1794863365189965 us; error 46 us; skew 500 ppm
I20261116 21:09:25.191201 3215 webserver.cc:577] Webserver started at http://127.0.117.254:44769/ using document root <none> and password file <none>
I20261116 21:09:25.191429 3215 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:25.191493 3215 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:25.191601 3215 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:25.192459 3215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c5ffe2f8ef8547e3ab544b370ff8db23"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.192808 3215 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c5ffe2f8ef8547e3ab544b370ff8db23"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.194103 3215 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20261116 21:09:25.194912 3229 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.195142 3215 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:25.195254 3215 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c5ffe2f8ef8547e3ab544b370ff8db23"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.195334 3215 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:25.220659 3215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:25.220978 3215 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:25.221127 3215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:25.225445 3215 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:36535
I20261116 21:09:25.225519 3281 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:36535 every 8 connection(s)
I20261116 21:09:25.225864 3215 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261116 21:09:25.226100 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3215
I20261116 21:09:25.226207 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261116 21:09:25.226498 3282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:25.229956 3282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23: Bootstrap starting.
I20261116 21:09:25.230672 3282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:25.230988 3282 log.cc:826] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:25.231704 3282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23: No bootstrap required, opened a new log
I20261116 21:09:25.232796 3282 raft_consensus.cc:360] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } }
I20261116 21:09:25.232934 3282 raft_consensus.cc:386] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:25.232978 3282 raft_consensus.cc:748] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ffe2f8ef8547e3ab544b370ff8db23, State: Initialized, Role: FOLLOWER
I20261116 21:09:25.233083 3282 consensus_queue.cc:261] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } }
I20261116 21:09:25.233152 3282 raft_consensus.cc:400] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:25.233192 3282 raft_consensus.cc:499] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:25.233238 3282 raft_consensus.cc:3096] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:25.233819 3282 raft_consensus.cc:521] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } }
I20261116 21:09:25.233958 3282 leader_election.cc:304] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5ffe2f8ef8547e3ab544b370ff8db23; no voters:
I20261116 21:09:25.234123 3282 leader_election.cc:290] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:25.234169 3287 raft_consensus.cc:2826] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:25.234337 3287 raft_consensus.cc:705] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [term 1 LEADER]: Becoming Leader. State: Replica: c5ffe2f8ef8547e3ab544b370ff8db23, State: Running, Role: LEADER
I20261116 21:09:25.234361 3282 sys_catalog.cc:565] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:25.234519 3287 consensus_queue.cc:238] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } }
I20261116 21:09:25.234929 3288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } } }
I20261116 21:09:25.235174 3288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:25.235109 3289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ffe2f8ef8547e3ab544b370ff8db23. Latest consensus state: current_term: 1 leader_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5ffe2f8ef8547e3ab544b370ff8db23" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 36535 } } }
I20261116 21:09:25.235334 3289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:25.235523 3297 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:25.235987 3297 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:25.237534 3297 catalog_manager.cc:1398] Generated new cluster ID: 5f68d76830f74fa0a97c89e9d6a3b1d1
I20261116 21:09:25.237596 3297 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:25.243764 3297 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:25.244338 3297 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:25.259095 3297 catalog_manager.cc:6221] T 00000000000000000000000000000000 P c5ffe2f8ef8547e3ab544b370ff8db23: Generated new TSK 0
I20261116 21:09:25.259317 3297 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:25.262642 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:25.342348 3306 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:25.342536 3306 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:25.342562 3306 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:25.342603 3306 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:25.344148 3306 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:25.344275 3306 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:25.344303 3306 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:25.344339 3306 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:25.344393 3306 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:25.346081 3306 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--never_fsync=true
--heap_profile_path=/tmp/kudu.3306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:25.346319 3306 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:25.346505 3306 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:25.349074 3311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.349074 3314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.349229 3306 server_base.cc:1061] running on GCE node
W20261116 21:09:25.349421 3312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.349623 3306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:25.349851 3306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:25.351009 3306 hybrid_clock.cc:648] HybridClock initialized: now 1794863365350990 us; error 44 us; skew 500 ppm
I20261116 21:09:25.352171 3306 webserver.cc:577] Webserver started at http://127.0.117.193:37307/ using document root <none> and password file <none>
I20261116 21:09:25.352380 3306 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:25.352439 3306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:25.352545 3306 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:25.353492 3306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ca23250a0e4a43678660112e28335012"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.353816 3306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ca23250a0e4a43678660112e28335012"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.354996 3306 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:25.355813 3320 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.355990 3306 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:25.356066 3306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ca23250a0e4a43678660112e28335012"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.356143 3306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:25.366276 3306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:25.366535 3306 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:25.366663 3306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:25.366882 3306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:25.367208 3306 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:25.367295 3306 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.367341 3306 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:25.367370 3306 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.373482 3306 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:38011
I20261116 21:09:25.373524 3433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:38011 every 8 connection(s)
I20261116 21:09:25.373847 3306 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261116 21:09:25.378137 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3306
I20261116 21:09:25.378285 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261116 21:09:25.379096 3434 heartbeater.cc:345] Connected to a master server at 127.0.117.254:36535
I20261116 21:09:25.379204 3434 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:25.379400 3434 heartbeater.cc:508] Master 127.0.117.254:36535 requested a full tablet report, sending...
I20261116 21:09:25.379509 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261116 21:09:25.380059 3246 ts_manager.cc:210] Registered new tserver with Master: ca23250a0e4a43678660112e28335012 (127.0.117.193:38011)
I20261116 21:09:25.381017 3246 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:52319
W20261116 21:09:25.464037 3437 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:25.464190 3437 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:25.464206 3437 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:25.464222 3437 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:25.465904 3437 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:25.465950 3437 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:25.465966 3437 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:25.465988 3437 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:25.466023 3437 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:25.467626 3437 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--never_fsync=true
--heap_profile_path=/tmp/kudu.3437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:25.467859 3437 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:25.468091 3437 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:25.470544 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.470561 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.470548 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.471114 3437 server_base.cc:1061] running on GCE node
I20261116 21:09:25.471303 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:25.471505 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:25.472704 3437 hybrid_clock.cc:648] HybridClock initialized: now 1794863365472687 us; error 30 us; skew 500 ppm
I20261116 21:09:25.473858 3437 webserver.cc:577] Webserver started at http://127.0.117.194:33057/ using document root <none> and password file <none>
I20261116 21:09:25.474066 3437 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:25.474128 3437 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:25.474255 3437 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:25.475057 3437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "91b045b52e26481cb7ab3fa29c5788a5"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.475322 3437 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "91b045b52e26481cb7ab3fa29c5788a5"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.476531 3437 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20261116 21:09:25.477171 3451 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.477415 3437 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.477468 3437 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "91b045b52e26481cb7ab3fa29c5788a5"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.477509 3437 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:25.492568 3437 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:25.492746 3437 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:25.492836 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:25.492977 3437 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:25.493305 3437 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:25.493397 3437 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.493422 3437 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:25.493466 3437 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.499662 3437 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:33933
I20261116 21:09:25.499701 3564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:33933 every 8 connection(s)
I20261116 21:09:25.500028 3437 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261116 21:09:25.504586 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3437
I20261116 21:09:25.504653 3565 heartbeater.cc:345] Connected to a master server at 127.0.117.254:36535
I20261116 21:09:25.504703 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261116 21:09:25.504752 3565 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:25.504940 3565 heartbeater.cc:508] Master 127.0.117.254:36535 requested a full tablet report, sending...
I20261116 21:09:25.505440 3241 ts_manager.cc:210] Registered new tserver with Master: 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:25.505888 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261116 21:09:25.505918 3241 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:58509
W20261116 21:09:25.585584 3568 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:25.585742 3568 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:25.585759 3568 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:25.585778 3568 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:25.587280 3568 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:25.587332 3568 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:25.587349 3568 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:25.587373 3568 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:25.587387 3568 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:25.588939 3568 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--never_fsync=true
--heap_profile_path=/tmp/kudu.3568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:25.589113 3568 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:25.589263 3568 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:25.591673 3574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.591708 3576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.591682 3573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.591840 3568 server_base.cc:1061] running on GCE node
I20261116 21:09:25.592053 3568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:25.592260 3568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:25.593369 3568 hybrid_clock.cc:648] HybridClock initialized: now 1794863365593357 us; error 25 us; skew 500 ppm
I20261116 21:09:25.594508 3568 webserver.cc:577] Webserver started at http://127.0.117.195:43027/ using document root <none> and password file <none>
I20261116 21:09:25.594723 3568 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:25.594792 3568 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:25.594903 3568 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:25.595707 3568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ebc762516f22435b9fd806f513cb0b21"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.596058 3568 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ebc762516f22435b9fd806f513cb0b21"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.597183 3568 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20261116 21:09:25.597994 3582 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.598186 3568 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:25.598263 3568 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ebc762516f22435b9fd806f513cb0b21"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.598320 3568 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:25.609069 3568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:25.609354 3568 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:25.609474 3568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:25.609671 3568 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:25.610018 3568 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:25.610064 3568 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.610105 3568 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:25.610132 3568 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.615818 3568 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:39899
I20261116 21:09:25.615857 3695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:39899 every 8 connection(s)
I20261116 21:09:25.616156 3568 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261116 21:09:25.620344 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3568
I20261116 21:09:25.620379 3696 heartbeater.cc:345] Connected to a master server at 127.0.117.254:36535
I20261116 21:09:25.620455 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261116 21:09:25.620507 3696 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:25.620724 3696 heartbeater.cc:508] Master 127.0.117.254:36535 requested a full tablet report, sending...
I20261116 21:09:25.621145 3246 ts_manager.cc:210] Registered new tserver with Master: ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899)
I20261116 21:09:25.621675 3246 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:51007
I20261116 21:09:25.621933 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.196:0
--local_ip_for_outbound_sockets=127.0.117.196
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:25.699219 3699 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:25.699391 3699 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:25.699416 3699 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:25.699450 3699 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:25.700959 3699 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:25.701025 3699 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:25.701069 3699 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:25.701112 3699 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:25.701141 3699 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.196
I20261116 21:09:25.702806 3699 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:36535
--never_fsync=true
--heap_profile_path=/tmp/kudu.3699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.196
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:25.703027 3699 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:25.703219 3699 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:25.705598 3704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.705672 3699 server_base.cc:1061] running on GCE node
W20261116 21:09:25.705633 3705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:25.705657 3707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:25.705914 3699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:25.706103 3699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:25.707280 3699 hybrid_clock.cc:648] HybridClock initialized: now 1794863365707250 us; error 46 us; skew 500 ppm
I20261116 21:09:25.708459 3699 webserver.cc:577] Webserver started at http://127.0.117.196:36097/ using document root <none> and password file <none>
I20261116 21:09:25.708670 3699 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:25.708712 3699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:25.708823 3699 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:25.709738 3699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b7ad724c4a2d441f896089e690bc90c9"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.710076 3699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b7ad724c4a2d441f896089e690bc90c9"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.711242 3699 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:25.711988 3713 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.712189 3699 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:25.712266 3699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b7ad724c4a2d441f896089e690bc90c9"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:25.712345 3699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:25.726938 3699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:25.727196 3699 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:25.727329 3699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:25.727530 3699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:25.727859 3699 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:25.727911 3699 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.727952 3699 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:25.727979 3699 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:25.733723 3699 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.196:35165
I20261116 21:09:25.733798 3826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.196:35165 every 8 connection(s)
I20261116 21:09:25.734129 3699 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261116 21:09:25.736081 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3699
I20261116 21:09:25.736176 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261116 21:09:25.738731 3827 heartbeater.cc:345] Connected to a master server at 127.0.117.254:36535
I20261116 21:09:25.738848 3827 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:25.739032 3827 heartbeater.cc:508] Master 127.0.117.254:36535 requested a full tablet report, sending...
I20261116 21:09:25.739410 3246 ts_manager.cc:210] Registered new tserver with Master: b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196:35165)
I20261116 21:09:25.739756 3246 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.196:52773
I20261116 21:09:25.748373 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:25.754092 3246 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36022:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261116 21:09:25.762050 3499 tablet_service.cc:1500] Processing CreateTablet for tablet 88394f59909542f7bc3604f558a73817 (DEFAULT_TABLE table=TestTable [id=404b1de165fe46d9a147d76032b764d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:25.762399 3499 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88394f59909542f7bc3604f558a73817. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:25.762450 3368 tablet_service.cc:1500] Processing CreateTablet for tablet 88394f59909542f7bc3604f558a73817 (DEFAULT_TABLE table=TestTable [id=404b1de165fe46d9a147d76032b764d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:25.762619 3630 tablet_service.cc:1500] Processing CreateTablet for tablet 88394f59909542f7bc3604f558a73817 (DEFAULT_TABLE table=TestTable [id=404b1de165fe46d9a147d76032b764d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:25.762725 3368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88394f59909542f7bc3604f558a73817. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:25.762873 3630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88394f59909542f7bc3604f558a73817. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:25.765090 3842 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Bootstrap starting.
I20261116 21:09:25.765534 3841 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Bootstrap starting.
I20261116 21:09:25.765573 3843 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Bootstrap starting.
I20261116 21:09:25.766062 3842 tablet_bootstrap.cc:654] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:25.766157 3843 tablet_bootstrap.cc:654] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:25.766176 3841 tablet_bootstrap.cc:654] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:25.766363 3842 log.cc:826] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:25.766412 3841 log.cc:826] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:25.766460 3843 log.cc:826] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:25.767009 3842 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: No bootstrap required, opened a new log
I20261116 21:09:25.767051 3843 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: No bootstrap required, opened a new log
I20261116 21:09:25.767078 3842 ts_tablet_manager.cc:1449] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:25.767103 3843 ts_tablet_manager.cc:1449] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:25.767109 3841 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: No bootstrap required, opened a new log
I20261116 21:09:25.767167 3841 ts_tablet_manager.cc:1449] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:25.768121 3843 raft_consensus.cc:360] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.768215 3843 raft_consensus.cc:386] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:25.768234 3843 raft_consensus.cc:748] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc762516f22435b9fd806f513cb0b21, State: Initialized, Role: FOLLOWER
I20261116 21:09:25.768330 3843 consensus_queue.cc:261] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.768549 3843 ts_tablet_manager.cc:1480] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:25.768662 3696 heartbeater.cc:500] Master 127.0.117.254:36535 was elected leader, sending a full tablet report...
I20261116 21:09:25.768631 3842 raft_consensus.cc:360] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.768764 3842 raft_consensus.cc:386] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:25.768795 3842 raft_consensus.cc:748] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b045b52e26481cb7ab3fa29c5788a5, State: Initialized, Role: FOLLOWER
I20261116 21:09:25.768868 3842 consensus_queue.cc:261] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.769089 3842 ts_tablet_manager.cc:1480] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:25.769198 3565 heartbeater.cc:500] Master 127.0.117.254:36535 was elected leader, sending a full tablet report...
I20261116 21:09:25.768671 3841 raft_consensus.cc:360] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.769418 3841 raft_consensus.cc:386] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:25.769474 3841 raft_consensus.cc:748] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca23250a0e4a43678660112e28335012, State: Initialized, Role: FOLLOWER
I20261116 21:09:25.769554 3841 consensus_queue.cc:261] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.769825 3841 ts_tablet_manager.cc:1480] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20261116 21:09:25.769867 3434 heartbeater.cc:500] Master 127.0.117.254:36535 was elected leader, sending a full tablet report...
W20261116 21:09:25.774828 3435 tablet.cc:2441] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 21:09:25.797506 3848 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:25.797618 3848 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.797998 3848 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899), ca23250a0e4a43678660112e28335012 (127.0.117.193:38011)
I20261116 21:09:25.800661 3388 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "91b045b52e26481cb7ab3fa29c5788a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012" is_pre_election: true
I20261116 21:09:25.800845 3388 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b045b52e26481cb7ab3fa29c5788a5 in term 0.
W20261116 21:09:25.800940 3566 tablet.cc:2441] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 21:09:25.801064 3453 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b045b52e26481cb7ab3fa29c5788a5, ca23250a0e4a43678660112e28335012; no voters:
I20261116 21:09:25.801170 3848 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 21:09:25.801222 3848 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:25.801163 3650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "91b045b52e26481cb7ab3fa29c5788a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc762516f22435b9fd806f513cb0b21" is_pre_election: true
I20261116 21:09:25.801239 3848 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:25.801357 3650 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91b045b52e26481cb7ab3fa29c5788a5 in term 0.
I20261116 21:09:25.801986 3848 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.802145 3848 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [CANDIDATE]: Term 1 election: Requested vote from peers ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899), ca23250a0e4a43678660112e28335012 (127.0.117.193:38011)
I20261116 21:09:25.802310 3650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "91b045b52e26481cb7ab3fa29c5788a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebc762516f22435b9fd806f513cb0b21"
I20261116 21:09:25.802371 3650 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:25.803401 3650 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b045b52e26481cb7ab3fa29c5788a5 in term 1.
I20261116 21:09:25.803500 3388 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "91b045b52e26481cb7ab3fa29c5788a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012"
I20261116 21:09:25.803599 3388 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:25.803591 3452 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91b045b52e26481cb7ab3fa29c5788a5, ebc762516f22435b9fd806f513cb0b21; no voters:
I20261116 21:09:25.803709 3848 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:25.803910 3848 raft_consensus.cc:705] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 1 LEADER]: Becoming Leader. State: Replica: 91b045b52e26481cb7ab3fa29c5788a5, State: Running, Role: LEADER
I20261116 21:09:25.804067 3848 consensus_queue.cc:238] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:25.804811 3388 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b045b52e26481cb7ab3fa29c5788a5 in term 1.
I20261116 21:09:25.805222 3848 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:25.805419 3848 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:25.805359 3246 catalog_manager.cc:5840] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194). New cstate: current_term: 1 leader_uuid: "91b045b52e26481cb7ab3fa29c5788a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } health_report { overall_health: HEALTHY } } }
I20261116 21:09:25.816756 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:25.817276 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ca23250a0e4a43678660112e28335012 to finish bootstrapping
I20261116 21:09:25.820674 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 91b045b52e26481cb7ab3fa29c5788a5 to finish bootstrapping
I20261116 21:09:25.823763 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ebc762516f22435b9fd806f513cb0b21 to finish bootstrapping
I20261116 21:09:25.826740 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3437
I20261116 21:09:26.740527 3827 heartbeater.cc:500] Master 127.0.117.254:36535 was elected leader, sending a full tablet report...
I20261116 21:09:27.307545 3862 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91b045b52e26481cb7ab3fa29c5788a5)
I20261116 21:09:27.307682 3862 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:27.307934 3863 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 91b045b52e26481cb7ab3fa29c5788a5)
I20261116 21:09:27.308046 3863 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:27.308300 3862 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
W20261116 21:09:27.308435 3322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:27.308763 3863 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca23250a0e4a43678660112e28335012 (127.0.117.193:38011), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
W20261116 21:09:27.309060 3322 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
W20261116 21:09:27.309029 3584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
W20261116 21:09:27.309527 3584 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:27.311278 3650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ca23250a0e4a43678660112e28335012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ebc762516f22435b9fd806f513cb0b21" is_pre_election: true
I20261116 21:09:27.311378 3650 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca23250a0e4a43678660112e28335012 in term 1.
I20261116 21:09:27.311417 3387 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ebc762516f22435b9fd806f513cb0b21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012" is_pre_election: true
I20261116 21:09:27.311525 3387 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc762516f22435b9fd806f513cb0b21 in term 1.
I20261116 21:09:27.311568 3321 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca23250a0e4a43678660112e28335012, ebc762516f22435b9fd806f513cb0b21; no voters: 91b045b52e26481cb7ab3fa29c5788a5
I20261116 21:09:27.311661 3584 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca23250a0e4a43678660112e28335012, ebc762516f22435b9fd806f513cb0b21; no voters: 91b045b52e26481cb7ab3fa29c5788a5
I20261116 21:09:27.311682 3862 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 21:09:27.311721 3862 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91b045b52e26481cb7ab3fa29c5788a5)
I20261116 21:09:27.311770 3863 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 21:09:27.311786 3862 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:27.311818 3863 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 91b045b52e26481cb7ab3fa29c5788a5)
I20261116 21:09:27.311841 3863 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:27.312402 3863 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:27.312409 3862 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:27.312527 3863 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 election: Requested vote from peers ca23250a0e4a43678660112e28335012 (127.0.117.193:38011), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:27.312736 3862 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 election: Requested vote from peers ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:27.312741 3387 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ebc762516f22435b9fd806f513cb0b21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012"
I20261116 21:09:27.312832 3650 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ca23250a0e4a43678660112e28335012" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ebc762516f22435b9fd806f513cb0b21"
I20261116 21:09:27.312873 3387 raft_consensus.cc:2415] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ebc762516f22435b9fd806f513cb0b21 in current term 2: Already voted for candidate ca23250a0e4a43678660112e28335012 in this term.
I20261116 21:09:27.312912 3650 raft_consensus.cc:2415] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca23250a0e4a43678660112e28335012 in current term 2: Already voted for candidate ebc762516f22435b9fd806f513cb0b21 in this term.
W20261116 21:09:27.313241 3322 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:27.313311 3322 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca23250a0e4a43678660112e28335012; no voters: 91b045b52e26481cb7ab3fa29c5788a5, ebc762516f22435b9fd806f513cb0b21
W20261116 21:09:27.313361 3584 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:27.313400 3862 raft_consensus.cc:2771] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261116 21:09:27.313408 3584 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ebc762516f22435b9fd806f513cb0b21; no voters: 91b045b52e26481cb7ab3fa29c5788a5, ca23250a0e4a43678660112e28335012
I20261116 21:09:27.313495 3863 raft_consensus.cc:2771] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261116 21:09:29.391219 3875 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:29.391338 3875 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:29.391567 3875 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca23250a0e4a43678660112e28335012 (127.0.117.193:38011), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:29.391911 3387 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ebc762516f22435b9fd806f513cb0b21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012" is_pre_election: true
I20261116 21:09:29.392025 3387 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebc762516f22435b9fd806f513cb0b21 in term 2.
I20261116 21:09:29.392423 3584 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca23250a0e4a43678660112e28335012, ebc762516f22435b9fd806f513cb0b21; no voters:
W20261116 21:09:29.392519 3584 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:29.392534 3875 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20261116 21:09:29.392567 3875 raft_consensus.cc:499] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:29.392588 3875 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 2 FOLLOWER]: Advancing to term 3
I20261116 21:09:29.393200 3875 raft_consensus.cc:521] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:29.393333 3875 leader_election.cc:290] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 election: Requested vote from peers ca23250a0e4a43678660112e28335012 (127.0.117.193:38011), 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:29.393769 3387 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "88394f59909542f7bc3604f558a73817" candidate_uuid: "ebc762516f22435b9fd806f513cb0b21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca23250a0e4a43678660112e28335012"
I20261116 21:09:29.393859 3387 raft_consensus.cc:3096] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 2 FOLLOWER]: Advancing to term 3
W20261116 21:09:29.394098 3584 leader_election.cc:336] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:29.394619 3387 raft_consensus.cc:2490] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebc762516f22435b9fd806f513cb0b21 in term 3.
I20261116 21:09:29.394780 3584 leader_election.cc:304] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca23250a0e4a43678660112e28335012, ebc762516f22435b9fd806f513cb0b21; no voters: 91b045b52e26481cb7ab3fa29c5788a5
I20261116 21:09:29.394883 3875 raft_consensus.cc:2826] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 FOLLOWER]: Leader election won for term 3
I20261116 21:09:29.394999 3875 raft_consensus.cc:705] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 LEADER]: Becoming Leader. State: Replica: ebc762516f22435b9fd806f513cb0b21, State: Running, Role: LEADER
I20261116 21:09:29.395103 3875 consensus_queue.cc:238] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:29.395817 3387 raft_consensus.cc:1285] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Refusing update from remote peer ebc762516f22435b9fd806f513cb0b21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261116 21:09:29.395949 3241 catalog_manager.cc:5840] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 reported cstate change: term changed from 1 to 3, leader changed from 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194) to ebc762516f22435b9fd806f513cb0b21 (127.0.117.195). New cstate: current_term: 3 leader_uuid: "ebc762516f22435b9fd806f513cb0b21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } health_report { overall_health: UNKNOWN } } }
I20261116 21:09:29.396131 3878 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20261116 21:09:29.396260 3584 consensus_peers.cc:597] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 -> Peer 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): Couldn't send request to peer 91b045b52e26481cb7ab3fa29c5788a5. Status: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:29.464367 471 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20261116 21:09:29.464675 471 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20261116 21:09:29.507210 471 raft_consensus-itest-base.cc:274] Waiting for log GC on ebc762516f22435b9fd806f513cb0b21
I20261116 21:09:29.524510 3697 maintenance_manager.cc:419] P ebc762516f22435b9fd806f513cb0b21: Scheduling FlushMRSOp(88394f59909542f7bc3604f558a73817): 4602312 bytes log retention, and flush 6119689 bytes memory
I20261116 21:09:29.529951 3587 maintenance_manager.cc:643] P ebc762516f22435b9fd806f513cb0b21: FlushMRSOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.005s user 0.005s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":174,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":737,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":215,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":216,"threads_started":2}
I20261116 21:09:29.530164 3697 maintenance_manager.cc:419] P ebc762516f22435b9fd806f513cb0b21: Scheduling LogGCOp(88394f59909542f7bc3604f558a73817): free 2761362 bytes of WAL
I20261116 21:09:29.530324 3587 log_reader.cc:385] T 88394f59909542f7bc3604f558a73817: removed 3 log segments from log reader
I20261116 21:09:29.530387 3587 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000001 (ops 1-9)
I20261116 21:09:29.530412 3587 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000002 (ops 10-16)
I20261116 21:09:29.530431 3587 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000003 (ops 17-23)
I20261116 21:09:29.531302 3587 maintenance_manager.cc:643] P ebc762516f22435b9fd806f513cb0b21: LogGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:29.531504 3697 maintenance_manager.cc:419] P ebc762516f22435b9fd806f513cb0b21: Scheduling UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817): 547 bytes on disk
I20261116 21:09:29.531759 3587 maintenance_manager.cc:643] P ebc762516f22435b9fd806f513cb0b21: UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4}
I20261116 21:09:29.537887 471 raft_consensus-itest-base.cc:279] Log GC complete on ebc762516f22435b9fd806f513cb0b21
I20261116 21:09:29.583375 3435 maintenance_manager.cc:419] P ca23250a0e4a43678660112e28335012: Scheduling FlushMRSOp(88394f59909542f7bc3604f558a73817): 4076220 bytes log retention, and flush 6119689 bytes memory
I20261116 21:09:29.589811 3325 maintenance_manager.cc:643] P ca23250a0e4a43678660112e28335012: FlushMRSOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.006s user 0.001s sys 0.005s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":178,"dirs.queue_time_us":189,"drs_written":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4,"lbm_write_time_us":300,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":37,"thread_start_us":234,"threads_started":2}
I20261116 21:09:29.590099 3435 maintenance_manager.cc:419] P ca23250a0e4a43678660112e28335012: Scheduling LogGCOp(88394f59909542f7bc3604f558a73817): free 2366875 bytes of WAL
I20261116 21:09:29.590229 3325 log_reader.cc:385] T 88394f59909542f7bc3604f558a73817: removed 3 log segments from log reader
I20261116 21:09:29.590266 3325 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000001 (ops 1-8)
I20261116 21:09:29.590330 3325 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000002 (ops 9-14)
I20261116 21:09:29.590377 3325 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000003 (ops 15-20)
I20261116 21:09:29.591205 3325 maintenance_manager.cc:643] P ca23250a0e4a43678660112e28335012: LogGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:29.591310 3435 maintenance_manager.cc:419] P ca23250a0e4a43678660112e28335012: Scheduling UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817): 547 bytes on disk
I20261116 21:09:29.591471 3325 maintenance_manager.cc:643] P ca23250a0e4a43678660112e28335012: UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20261116 21:09:29.798300 3909 consensus_queue.cc:754] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: The logs necessary to catch up peer 91b045b52e26481cb7ab3fa29c5788a5 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..32: Segment 1 which contained index 2 has been GCed)
I20261116 21:09:29.798393 3909 consensus_queue.cc:580] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: The logs necessary to catch up peer 91b045b52e26481cb7ab3fa29c5788a5 have been garbage collected. The replica will never be able to catch up
I20261116 21:09:29.799635 3650 consensus_queue.cc:238] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 3.39, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } }
I20261116 21:09:29.800297 3387 raft_consensus.cc:1285] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Refusing update from remote peer ebc762516f22435b9fd806f513cb0b21: Log matching property violated. Preceding OpId in replica: term: 3 index: 39. Preceding OpId from leader: term: 3 index: 40. (index mismatch)
I20261116 21:09:29.800603 3909 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20261116 21:09:29.801445 3878 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 LEADER]: Committing config change with OpId 3.40: config changed from index -1 to 40, VOTER 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } }
I20261116 21:09:29.801599 3387 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Committing config change with OpId 3.40: config changed from index -1 to 40, VOTER 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194) evicted. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } }
I20261116 21:09:29.802772 3230 catalog_manager.cc:5247] ChangeConfig:REMOVE_PEER RPC for tablet 88394f59909542f7bc3604f558a73817 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20261116 21:09:29.802857 3241 catalog_manager.cc:5840] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 reported cstate change: config changed from index -1 to 40, VOTER 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194) evicted. New cstate: current_term: 3 leader_uuid: "ebc762516f22435b9fd806f513cb0b21" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } }
I20261116 21:09:29.804018 3650 consensus_queue.cc:238] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 3.40, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } }
W20261116 21:09:29.804324 3231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:29.804524 3387 raft_consensus.cc:1285] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Refusing update from remote peer ebc762516f22435b9fd806f513cb0b21: Log matching property violated. Preceding OpId in replica: term: 3 index: 40. Preceding OpId from leader: term: 3 index: 41. (index mismatch)
I20261116 21:09:29.804845 3909 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 40, Time since last communication: 0.000s
W20261116 21:09:29.805025 3231 catalog_manager.cc:4792] TS 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 88394f59909542f7bc3604f558a73817: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:29.805560 3880 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 LEADER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } } }
I20261116 21:09:29.805744 3387 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Committing config change with OpId 3.41: config changed from index 40 to 41, NON_VOTER b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) added. New config: { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } } }
I20261116 21:09:29.806528 3230 catalog_manager.cc:5247] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 88394f59909542f7bc3604f558a73817 with cas_config_opid_index 40: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20261116 21:09:29.806924 3246 catalog_manager.cc:5840] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 reported cstate change: config changed from index 40 to 41, NON_VOTER b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) added. New cstate: current_term: 3 leader_uuid: "ebc762516f22435b9fd806f513cb0b21" committed_config { opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20261116 21:09:29.808982 3585 consensus_peers.cc:597] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 -> Peer b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196:35165): Couldn't send request to peer b7ad724c4a2d441f896089e690bc90c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88394f59909542f7bc3604f558a73817. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:30.350884 3918 ts_tablet_manager.cc:979] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Initiating tablet copy from peer ebc762516f22435b9fd806f513cb0b21 (127.0.117.195:39899)
I20261116 21:09:30.351116 3918 tablet_copy_client.cc:323] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.117.195:39899
I20261116 21:09:30.354012 3670 tablet_copy_service.cc:140] P ebc762516f22435b9fd806f513cb0b21: Received BeginTabletCopySession request for tablet 88394f59909542f7bc3604f558a73817 from peer b7ad724c4a2d441f896089e690bc90c9 ({username='slave'} at 127.0.117.196:37149)
I20261116 21:09:30.354089 3670 tablet_copy_service.cc:161] P ebc762516f22435b9fd806f513cb0b21: Beginning new tablet copy session on tablet 88394f59909542f7bc3604f558a73817 from peer b7ad724c4a2d441f896089e690bc90c9 at {username='slave'} at 127.0.117.196:37149: session id = b7ad724c4a2d441f896089e690bc90c9-88394f59909542f7bc3604f558a73817
I20261116 21:09:30.354759 3670 tablet_copy_source_session.cc:215] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Tablet Copy: opened 5 blocks and 3 log segments
I20261116 21:09:30.355647 3918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88394f59909542f7bc3604f558a73817. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:30.357720 3918 tablet_copy_client.cc:806] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: tablet copy: Starting download of 5 data blocks...
I20261116 21:09:30.361080 3918 tablet_copy_client.cc:670] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: tablet copy: Starting download of 3 WAL segments...
I20261116 21:09:30.367908 3918 tablet_copy_client.cc:538] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20261116 21:09:30.368788 3918 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Bootstrap starting.
I20261116 21:09:30.370499 3918 log.cc:826] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:30.373270 3918 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=5} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates
I20261116 21:09:30.375595 3918 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=12 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 2 replicates
I20261116 21:09:30.376731 3918 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Bootstrap replayed 3/3 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=16} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=3. Pending: 0 replicates
I20261116 21:09:30.376952 3918 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Bootstrap complete.
I20261116 21:09:30.377487 3918 ts_tablet_manager.cc:1449] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20261116 21:09:30.378262 3918 raft_consensus.cc:360] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } }
I20261116 21:09:30.378393 3918 raft_consensus.cc:748] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b7ad724c4a2d441f896089e690bc90c9, State: Initialized, Role: LEARNER
I20261116 21:09:30.378450 3918 consensus_queue.cc:261] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 3.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 41 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: NON_VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: true } }
I20261116 21:09:30.378669 3918 ts_tablet_manager.cc:1480] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:30.379171 3669 tablet_copy_service.cc:342] P ebc762516f22435b9fd806f513cb0b21: Request end of tablet copy session b7ad724c4a2d441f896089e690bc90c9-88394f59909542f7bc3604f558a73817 received from {username='slave'} at 127.0.117.196:37149
I20261116 21:09:30.379253 3669 tablet_copy_service.cc:434] P ebc762516f22435b9fd806f513cb0b21: ending tablet copy session b7ad724c4a2d441f896089e690bc90c9-88394f59909542f7bc3604f558a73817 on tablet 88394f59909542f7bc3604f558a73817 with peer b7ad724c4a2d441f896089e690bc90c9
I20261116 21:09:30.443140 3828 maintenance_manager.cc:419] P b7ad724c4a2d441f896089e690bc90c9: Scheduling UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817): 547 bytes on disk
I20261116 21:09:30.443588 3718 maintenance_manager.cc:643] P b7ad724c4a2d441f896089e690bc90c9: UndoDeltaBlockGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":22,"lbm_reads_lt_1ms":4}
I20261116 21:09:30.827287 3781 raft_consensus.cc:1227] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.40->[3.41-3.41] Dedup: 3.41->[]
I20261116 21:09:30.834774 3697 maintenance_manager.cc:419] P ebc762516f22435b9fd806f513cb0b21: Scheduling LogGCOp(88394f59909542f7bc3604f558a73817): free 1840950 bytes of WAL
I20261116 21:09:30.835074 3587 log_reader.cc:385] T 88394f59909542f7bc3604f558a73817: removed 2 log segments from log reader
I20261116 21:09:30.835166 3587 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000004 (ops 24-30)
I20261116 21:09:30.835193 3587 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000005 (ops 31-37)
I20261116 21:09:30.835781 3587 maintenance_manager.cc:643] P ebc762516f22435b9fd806f513cb0b21: LogGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
W20261116 21:09:31.012878 3231 catalog_manager.cc:4792] TS 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 88394f59909542f7bc3604f558a73817: Network error: Client connection negotiation failed: client connection to 127.0.117.194:33933: connect: Connection refused (error 111)
I20261116 21:09:31.295244 3435 maintenance_manager.cc:419] P ca23250a0e4a43678660112e28335012: Scheduling LogGCOp(88394f59909542f7bc3604f558a73817): free 1709345 bytes of WAL
I20261116 21:09:31.295496 3325 log_reader.cc:385] T 88394f59909542f7bc3604f558a73817: removed 2 log segments from log reader
I20261116 21:09:31.295569 3325 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000004 (ops 21-26)
I20261116 21:09:31.295590 3325 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000005 (ops 27-33)
I20261116 21:09:31.296152 3325 maintenance_manager.cc:643] P ca23250a0e4a43678660112e28335012: LogGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:31.321751 3926 raft_consensus.cc:1074] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21: attempting to promote NON_VOTER b7ad724c4a2d441f896089e690bc90c9 to VOTER
I20261116 21:09:31.322067 3926 consensus_queue.cc:238] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 3.41, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false } }
I20261116 21:09:31.322739 3387 raft_consensus.cc:1285] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Refusing update from remote peer ebc762516f22435b9fd806f513cb0b21: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20261116 21:09:31.322743 3781 raft_consensus.cc:1285] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [term 3 LEARNER]: Refusing update from remote peer ebc762516f22435b9fd806f513cb0b21: Log matching property violated. Preceding OpId in replica: term: 3 index: 41. Preceding OpId from leader: term: 3 index: 42. (index mismatch)
I20261116 21:09:31.323033 3926 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261116 21:09:31.323278 3917 consensus_queue.cc:1049] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20261116 21:09:31.323902 3926 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 [term 3 LEADER]: Committing config change with OpId 3.42: config changed from index 41 to 42, b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false } } }
I20261116 21:09:31.324045 3387 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P ca23250a0e4a43678660112e28335012 [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false } } }
I20261116 21:09:31.324483 3781 raft_consensus.cc:2983] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9 [term 3 FOLLOWER]: Committing config change with OpId 3.42: config changed from index 41 to 42, b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) changed from NON_VOTER to VOTER. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false } } }
I20261116 21:09:31.325251 3241 catalog_manager.cc:5840] T 88394f59909542f7bc3604f558a73817 P ebc762516f22435b9fd806f513cb0b21 reported cstate change: config changed from index 41 to 42, b7ad724c4a2d441f896089e690bc90c9 (127.0.117.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ebc762516f22435b9fd806f513cb0b21" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ad724c4a2d441f896089e690bc90c9" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 35165 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20261116 21:09:31.346192 3828 maintenance_manager.cc:419] P b7ad724c4a2d441f896089e690bc90c9: Scheduling LogGCOp(88394f59909542f7bc3604f558a73817): free 1840860 bytes of WAL
I20261116 21:09:31.346498 3718 log_reader.cc:385] T 88394f59909542f7bc3604f558a73817: removed 2 log segments from log reader
I20261116 21:09:31.346540 3718 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000001 (ops 24-30)
I20261116 21:09:31.346580 3718 log.cc:1079] T 88394f59909542f7bc3604f558a73817 P b7ad724c4a2d441f896089e690bc90c9: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/88394f59909542f7bc3604f558a73817/wal-000000002 (ops 31-37)
I20261116 21:09:31.347115 3718 maintenance_manager.cc:643] P b7ad724c4a2d441f896089e690bc90c9: LogGCOp(88394f59909542f7bc3604f558a73817) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:31.538748 471 raft_consensus-itest-base.cc:295] Servers converged with original term 3
I20261116 21:09:31.538818 471 raft_consensus-itest-base.cc:303] Restarting 91b045b52e26481cb7ab3fa29c5788a5
I20261116 21:09:31.539041 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:33933
--local_ip_for_outbound_sockets=127.0.117.194
--tserver_master_addrs=127.0.117.254:36535
--webserver_port=33057
--webserver_interface=127.0.117.194
--builtin_ntp_servers=127.0.117.212:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:31.619334 3938 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:31.619537 3938 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:31.619565 3938 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:31.619601 3938 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:31.621160 3938 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:31.621229 3938 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:31.621256 3938 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:31.621294 3938 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:31.621356 3938 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:31.622982 3938 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:33933
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=33057
--tserver_master_addrs=127.0.117.254:36535
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:31.623227 3938 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:31.623430 3938 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:31.626160 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:31.626161 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:31.626349 3938 server_base.cc:1061] running on GCE node
W20261116 21:09:31.626168 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:31.626623 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:31.626883 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:31.628055 3938 hybrid_clock.cc:648] HybridClock initialized: now 1794863371628040 us; error 30 us; skew 500 ppm
I20261116 21:09:31.629307 3938 webserver.cc:577] Webserver started at http://127.0.117.194:33057/ using document root <none> and password file <none>
I20261116 21:09:31.629580 3938 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:31.629658 3938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:31.630865 3938 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20261116 21:09:31.631645 3952 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:31.631829 3938 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20261116 21:09:31.631909 3938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "91b045b52e26481cb7ab3fa29c5788a5"
format_stamp: "Formatted at 2026-11-16 21:09:25 on dist-test-slave-gg0q"
I20261116 21:09:31.632196 3938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:31.639719 3938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:31.639971 3938 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:31.640095 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:31.640314 3938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:31.640694 3959 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261116 21:09:31.641532 3938 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261116 21:09:31.641590 3938 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:31.641654 3938 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261116 21:09:31.642174 3938 ts_tablet_manager.cc:662] Registered 1 tablets
I20261116 21:09:31.642268 3938 ts_tablet_manager.cc:641] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:31.642293 3959 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Bootstrap starting.
I20261116 21:09:31.645655 3959 log.cc:826] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:31.646890 3959 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 21:09:31.647217 3959 tablet_bootstrap.cc:492] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Bootstrap complete.
I20261116 21:09:31.647521 3959 ts_tablet_manager.cc:1449] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20261116 21:09:31.649247 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:33933
I20261116 21:09:31.649433 4067 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:33933 every 8 connection(s)
I20261116 21:09:31.649698 3938 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehind.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261116 21:09:31.649639 3959 raft_consensus.cc:360] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:31.650043 3959 raft_consensus.cc:748] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b045b52e26481cb7ab3fa29c5788a5, State: Initialized, Role: FOLLOWER
I20261116 21:09:31.650388 3959 consensus_queue.cc:261] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebc762516f22435b9fd806f513cb0b21" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 39899 } } peers { permanent_uuid: "ca23250a0e4a43678660112e28335012" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 38011 } } peers { permanent_uuid: "91b045b52e26481cb7ab3fa29c5788a5" member_type: VOTER last_known_addr { host: "127.0.117.194" port: 33933 } }
I20261116 21:09:31.650688 3959 ts_tablet_manager.cc:1480] T 88394f59909542f7bc3604f558a73817 P 91b045b52e26481cb7ab3fa29c5788a5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20261116 21:09:31.653934 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 3938
I20261116 21:09:31.655962 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3306
I20261116 21:09:31.659767 4068 heartbeater.cc:345] Connected to a master server at 127.0.117.254:36535
I20261116 21:09:31.659874 4068 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:31.660109 4068 heartbeater.cc:508] Master 127.0.117.254:36535 requested a full tablet report, sending...
I20261116 21:09:31.660696 3246 ts_manager.cc:210] Re-registered known tserver with Master: 91b045b52e26481cb7ab3fa29c5788a5 (127.0.117.194:33933)
I20261116 21:09:31.661126 3246 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:36479
I20261116 21:09:31.661761 4068 heartbeater.cc:500] Master 127.0.117.254:36535 was elected leader, sending a full tablet report...
I20261116 21:09:31.663254 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3938
I20261116 21:09:31.668227 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3568
I20261116 21:09:31.674350 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3699
I20261116 21:09:31.679493 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 3215
2026-11-16T21:09:31Z chronyd exiting
[ OK ] SlowTabletBootstrapTest.TestFallBehind (6607 ms)
[ RUN ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
I20261116 21:09:31.701762 471 ts_itest-base.cc:115] Starting cluster with:
I20261116 21:09:31.701831 471 ts_itest-base.cc:116] --------------
I20261116 21:09:31.701853 471 ts_itest-base.cc:117] 4 tablet servers
I20261116 21:09:31.701879 471 ts_itest-base.cc:118] 3 replicas per TS
I20261116 21:09:31.701903 471 ts_itest-base.cc:119] --------------
2026-11-16T21:09:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-11-16T21:09:31Z Disabled control of system clock
I20261116 21:09:31.709038 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:33845
--webserver_interface=127.0.117.254
--webserver_port=0
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.117.254:33845 with env {}
W20261116 21:09:31.786175 4080 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:31.786338 4080 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:31.786369 4080 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:31.787853 4080 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20261116 21:09:31.787922 4080 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:31.787945 4080 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20261116 21:09:31.787976 4080 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20261116 21:09:31.789592 4080 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.117.254:33845
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.117.254:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.117.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:31.789819 4080 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:31.790017 4080 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:31.792235 4088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:31.792240 4085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:31.792315 4086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:31.792572 4080 server_base.cc:1061] running on GCE node
I20261116 21:09:31.792789 4080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:31.793026 4080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:31.794224 4080 hybrid_clock.cc:648] HybridClock initialized: now 1794863371794211 us; error 24 us; skew 500 ppm
I20261116 21:09:31.795212 4080 webserver.cc:577] Webserver started at http://127.0.117.254:41377/ using document root <none> and password file <none>
I20261116 21:09:31.795437 4080 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:31.795501 4080 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:31.795605 4080 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:31.796443 4080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "456afe3bc7fd43aba84fa27e66389486"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.796779 4080 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "456afe3bc7fd43aba84fa27e66389486"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.798041 4080 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20261116 21:09:31.798688 4094 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:31.798842 4080 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20261116 21:09:31.798918 4080 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
uuid: "456afe3bc7fd43aba84fa27e66389486"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.798975 4080 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:31.814250 4080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:31.814551 4080 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:31.814695 4080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:31.818917 4080 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.254:33845
I20261116 21:09:31.818955 4146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.254:33845 every 8 connection(s)
I20261116 21:09:31.819288 4080 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20261116 21:09:31.819869 4147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:31.822201 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486: Bootstrap starting.
I20261116 21:09:31.822878 4147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:31.822902 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4080
I20261116 21:09:31.823031 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/master-0/wal/instance
I20261116 21:09:31.823156 4147 log.cc:826] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:31.823800 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486: No bootstrap required, opened a new log
I20261116 21:09:31.825070 4147 raft_consensus.cc:360] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } }
I20261116 21:09:31.825203 4147 raft_consensus.cc:386] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:31.825237 4147 raft_consensus.cc:748] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456afe3bc7fd43aba84fa27e66389486, State: Initialized, Role: FOLLOWER
I20261116 21:09:31.825362 4147 consensus_queue.cc:261] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } }
I20261116 21:09:31.825459 4147 raft_consensus.cc:400] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20261116 21:09:31.825502 4147 raft_consensus.cc:499] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20261116 21:09:31.825546 4147 raft_consensus.cc:3096] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:31.826076 4147 raft_consensus.cc:521] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } }
I20261116 21:09:31.826201 4147 leader_election.cc:304] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456afe3bc7fd43aba84fa27e66389486; no voters:
I20261116 21:09:31.826371 4147 leader_election.cc:290] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20261116 21:09:31.826678 4147 sys_catalog.cc:565] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [sys.catalog]: configured and running, proceeding with master startup.
I20261116 21:09:31.826402 4152 raft_consensus.cc:2826] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:31.826970 4152 raft_consensus.cc:705] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [term 1 LEADER]: Becoming Leader. State: Replica: 456afe3bc7fd43aba84fa27e66389486, State: Running, Role: LEADER
I20261116 21:09:31.827105 4152 consensus_queue.cc:238] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } }
I20261116 21:09:31.827497 4154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456afe3bc7fd43aba84fa27e66389486. Latest consensus state: current_term: 1 leader_uuid: "456afe3bc7fd43aba84fa27e66389486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } } }
I20261116 21:09:31.827616 4154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:31.827477 4153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456afe3bc7fd43aba84fa27e66389486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456afe3bc7fd43aba84fa27e66389486" member_type: VOTER last_known_addr { host: "127.0.117.254" port: 33845 } } }
I20261116 21:09:31.827785 4153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486 [sys.catalog]: This master's current role is: LEADER
I20261116 21:09:31.827952 4161 catalog_manager.cc:1526] Loading table and tablet metadata into memory...
I20261116 21:09:31.828529 4161 catalog_manager.cc:1535] Initializing Kudu cluster ID...
I20261116 21:09:31.829977 4161 catalog_manager.cc:1398] Generated new cluster ID: 220777a82a93404a869502260831a6c9
I20261116 21:09:31.830030 4161 catalog_manager.cc:1546] Initializing Kudu internal certificate authority...
I20261116 21:09:31.851985 4161 catalog_manager.cc:1421] Generated new certificate authority record
I20261116 21:09:31.852569 4161 catalog_manager.cc:1555] Loading token signing keys...
I20261116 21:09:31.862222 4161 catalog_manager.cc:6221] T 00000000000000000000000000000000 P 456afe3bc7fd43aba84fa27e66389486: Generated new TSK 0
I20261116 21:09:31.862432 4161 catalog_manager.cc:1565] Initializing in-progress tserver states...
I20261116 21:09:31.865598 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:0
--local_ip_for_outbound_sockets=127.0.117.193
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:31.943738 4171 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:31.943915 4171 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:31.943940 4171 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:31.943979 4171 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:31.945547 4171 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:31.945605 4171 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:31.945631 4171 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:31.945665 4171 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:31.945722 4171 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:31.947265 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:31.947484 4171 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:31.947686 4171 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:31.950029 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:31.950035 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:31.950071 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:31.950650 4171 server_base.cc:1061] running on GCE node
I20261116 21:09:31.950809 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:31.951018 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:31.952164 4171 hybrid_clock.cc:648] HybridClock initialized: now 1794863371952124 us; error 60 us; skew 500 ppm
I20261116 21:09:31.953305 4171 webserver.cc:577] Webserver started at http://127.0.117.193:33593/ using document root <none> and password file <none>
I20261116 21:09:31.953560 4171 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:31.953613 4171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:31.953711 4171 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:31.954716 4171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.954972 4171 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.956230 4171 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20261116 21:09:31.957017 4185 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:31.957207 4171 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:31.957288 4171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:31.957373 4171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:31.968564 4171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:31.968809 4171 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:31.968936 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:31.969130 4171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:31.969513 4171 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:31.969564 4171 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:31.969609 4171 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:31.969635 4171 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:31.975451 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:39387
I20261116 21:09:31.975486 4298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:39387 every 8 connection(s)
I20261116 21:09:31.975839 4171 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261116 21:09:31.980465 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4171
I20261116 21:09:31.980559 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20261116 21:09:31.981374 4299 heartbeater.cc:345] Connected to a master server at 127.0.117.254:33845
I20261116 21:09:31.981467 4299 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:31.981662 4299 heartbeater.cc:508] Master 127.0.117.254:33845 requested a full tablet report, sending...
I20261116 21:09:31.981830 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.194:0
--local_ip_for_outbound_sockets=127.0.117.194
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20261116 21:09:31.982132 4111 ts_manager.cc:210] Registered new tserver with Master: a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:31.982848 4111 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:43563
W20261116 21:09:32.060109 4302 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:32.060452 4302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:32.060496 4302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:32.060515 4302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:32.062186 4302 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:32.062234 4302 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:32.062251 4302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:32.062273 4302 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:32.062286 4302 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.194
I20261116 21:09:32.063900 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.117.194
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.194
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:32.064100 4302 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:32.064303 4302 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:32.066627 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.066671 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.066661 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:32.067147 4302 server_base.cc:1061] running on GCE node
I20261116 21:09:32.067306 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:32.067507 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:32.068655 4302 hybrid_clock.cc:648] HybridClock initialized: now 1794863372068649 us; error 39 us; skew 500 ppm
I20261116 21:09:32.069741 4302 webserver.cc:577] Webserver started at http://127.0.117.194:39645/ using document root <none> and password file <none>
I20261116 21:09:32.069952 4302 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:32.070009 4302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:32.070114 4302 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:32.071008 4302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d4287e3703c349bcb8e37d1965c81aba"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.071352 4302 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d4287e3703c349bcb8e37d1965c81aba"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.072495 4302 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20261116 21:09:32.073241 4316 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.073531 4302 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:32.073611 4302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d4287e3703c349bcb8e37d1965c81aba"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.073688 4302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:32.080823 4302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:32.081037 4302 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:32.081161 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:32.081379 4302 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:32.081702 4302 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:32.081754 4302 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.081795 4302 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:32.081851 4302 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.087251 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.194:41671
I20261116 21:09:32.087356 4429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.194:41671 every 8 connection(s)
I20261116 21:09:32.087632 4302 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20261116 21:09:32.092252 4430 heartbeater.cc:345] Connected to a master server at 127.0.117.254:33845
I20261116 21:09:32.092341 4430 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:32.092543 4430 heartbeater.cc:508] Master 127.0.117.254:33845 requested a full tablet report, sending...
I20261116 21:09:32.092926 4111 ts_manager.cc:210] Registered new tserver with Master: d4287e3703c349bcb8e37d1965c81aba (127.0.117.194:41671)
I20261116 21:09:32.093303 4111 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.194:58113
I20261116 21:09:32.096673 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4302
I20261116 21:09:32.096753 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20261116 21:09:32.097927 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.195:0
--local_ip_for_outbound_sockets=127.0.117.195
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:32.176195 4433 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:32.176362 4433 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:32.176422 4433 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:32.176592 4433 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:32.178272 4433 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:32.178320 4433 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:32.178336 4433 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:32.178359 4433 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:32.178400 4433 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.195
I20261116 21:09:32.179968 4433 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.117.195
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--never_fsync=true
--heap_profile_path=/tmp/kudu.4433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.195
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:32.180193 4433 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:32.180385 4433 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:32.182760 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.182694 4439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.182694 4441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:32.183046 4433 server_base.cc:1061] running on GCE node
I20261116 21:09:32.183214 4433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:32.183413 4433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:32.184574 4433 hybrid_clock.cc:648] HybridClock initialized: now 1794863372184550 us; error 39 us; skew 500 ppm
I20261116 21:09:32.185698 4433 webserver.cc:577] Webserver started at http://127.0.117.195:39949/ using document root <none> and password file <none>
I20261116 21:09:32.185930 4433 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:32.185990 4433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:32.186092 4433 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:32.186908 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.187265 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.188418 4433 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20261116 21:09:32.189129 4447 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.189361 4433 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:32.189446 4433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.189519 4433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:32.199149 4433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:32.199378 4433 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:32.199501 4433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:32.199692 4433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:32.199998 4433 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:32.200052 4433 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.200093 4433 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:32.200119 4433 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.205946 4433 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.195:42337
I20261116 21:09:32.206024 4560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.195:42337 every 8 connection(s)
I20261116 21:09:32.206336 4433 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20261116 21:09:32.210695 4561 heartbeater.cc:345] Connected to a master server at 127.0.117.254:33845
I20261116 21:09:32.210788 4561 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:32.210983 4561 heartbeater.cc:508] Master 127.0.117.254:33845 requested a full tablet report, sending...
I20261116 21:09:32.211369 4111 ts_manager.cc:210] Registered new tserver with Master: cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337)
I20261116 21:09:32.211710 4111 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.195:60725
I20261116 21:09:32.212260 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4433
I20261116 21:09:32.212340 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20261116 21:09:32.213478 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.196:0
--local_ip_for_outbound_sockets=127.0.117.196
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20261116 21:09:32.290014 4564 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:32.290154 4564 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:32.290169 4564 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:32.290185 4564 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:32.291684 4564 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:32.291724 4564 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:32.291739 4564 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:32.291791 4564 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:32.291818 4564 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.196
I20261116 21:09:32.293485 4564 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.117.196
--webserver_port=0
--tserver_master_addrs=127.0.117.254:33845
--never_fsync=true
--heap_profile_path=/tmp/kudu.4564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.196
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:32.293678 4564 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:32.293874 4564 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20261116 21:09:32.296289 4570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.296303 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.296383 4572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:32.296348 4564 server_base.cc:1061] running on GCE node
I20261116 21:09:32.296607 4564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:32.296814 4564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:32.297955 4564 hybrid_clock.cc:648] HybridClock initialized: now 1794863372297942 us; error 29 us; skew 500 ppm
I20261116 21:09:32.298997 4564 webserver.cc:577] Webserver started at http://127.0.117.196:43911/ using document root <none> and password file <none>
I20261116 21:09:32.299206 4564 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:32.299266 4564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:32.299368 4564 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20261116 21:09:32.300166 4564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c619addfb98242838dc21a5efb20039d"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.300498 4564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c619addfb98242838dc21a5efb20039d"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.301823 4564 fs_manager.cc:696] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20261116 21:09:32.302631 4578 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.302829 4564 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20261116 21:09:32.302903 4564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c619addfb98242838dc21a5efb20039d"
format_stamp: "Formatted at 2026-11-16 21:09:32 on dist-test-slave-gg0q"
I20261116 21:09:32.302956 4564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:32.334115 4564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:32.334365 4564 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:32.334492 4564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:32.334699 4564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:32.335035 4564 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20261116 21:09:32.335089 4564 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.335135 4564 ts_tablet_manager.cc:662] Registered 0 tablets
I20261116 21:09:32.335161 4564 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.340847 4564 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.196:40627
I20261116 21:09:32.340883 4691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.196:40627 every 8 connection(s)
I20261116 21:09:32.341207 4564 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20261116 21:09:32.345369 4692 heartbeater.cc:345] Connected to a master server at 127.0.117.254:33845
I20261116 21:09:32.345469 4692 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:32.345618 4692 heartbeater.cc:508] Master 127.0.117.254:33845 requested a full tablet report, sending...
I20261116 21:09:32.345942 4111 ts_manager.cc:210] Registered new tserver with Master: c619addfb98242838dc21a5efb20039d (127.0.117.196:40627)
I20261116 21:09:32.346325 4111 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.196:42705
I20261116 21:09:32.348615 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4564
I20261116 21:09:32.348718 471 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20261116 21:09:32.349794 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:32.355094 4111 catalog_manager.cc:2298] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59972:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20261116 21:09:32.362751 4626 tablet_service.cc:1500] Processing CreateTablet for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6 (DEFAULT_TABLE table=TestTable [id=228ff408d24e4da09d6024adbed6d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:32.363121 4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:32.363348 4233 tablet_service.cc:1500] Processing CreateTablet for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6 (DEFAULT_TABLE table=TestTable [id=228ff408d24e4da09d6024adbed6d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:32.363541 4233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:32.363564 4495 tablet_service.cc:1500] Processing CreateTablet for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6 (DEFAULT_TABLE table=TestTable [id=228ff408d24e4da09d6024adbed6d4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20261116 21:09:32.363806 4495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c4b46ee5b064b9fad4aded3b4fd1fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20261116 21:09:32.365396 4706 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Bootstrap starting.
I20261116 21:09:32.365836 4707 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Bootstrap starting.
I20261116 21:09:32.366057 4706 tablet_bootstrap.cc:654] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:32.366344 4708 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Bootstrap starting.
I20261116 21:09:32.366369 4706 log.cc:826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:32.366480 4707 tablet_bootstrap.cc:654] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:32.366796 4707 log.cc:826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:32.367043 4708 tablet_bootstrap.cc:654] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Neither blocks nor log segments found. Creating new log.
I20261116 21:09:32.367338 4708 log.cc:826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:32.367434 4707 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: No bootstrap required, opened a new log
I20261116 21:09:32.367507 4707 ts_tablet_manager.cc:1449] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:32.367957 4708 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: No bootstrap required, opened a new log
I20261116 21:09:32.368028 4708 ts_tablet_manager.cc:1449] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20261116 21:09:32.369079 4707 raft_consensus.cc:360] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.369225 4707 raft_consensus.cc:386] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:32.369256 4707 raft_consensus.cc:748] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c619addfb98242838dc21a5efb20039d, State: Initialized, Role: FOLLOWER
I20261116 21:09:32.369441 4707 consensus_queue.cc:261] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.369582 4708 raft_consensus.cc:360] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.369678 4706 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: No bootstrap required, opened a new log
I20261116 21:09:32.369680 4707 ts_tablet_manager.cc:1480] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20261116 21:09:32.369737 4692 heartbeater.cc:500] Master 127.0.117.254:33845 was elected leader, sending a full tablet report...
I20261116 21:09:32.369777 4708 raft_consensus.cc:386] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:32.369784 4706 ts_tablet_manager.cc:1449] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20261116 21:09:32.369813 4708 raft_consensus.cc:748] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd60873f44194f8d92ffc6a9fc4ac77c, State: Initialized, Role: FOLLOWER
I20261116 21:09:32.369911 4708 consensus_queue.cc:261] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.370155 4708 ts_tablet_manager.cc:1480] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:32.370287 4561 heartbeater.cc:500] Master 127.0.117.254:33845 was elected leader, sending a full tablet report...
I20261116 21:09:32.371417 4706 raft_consensus.cc:360] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.371513 4706 raft_consensus.cc:386] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20261116 21:09:32.371531 4706 raft_consensus.cc:748] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67bb3cfd7a645629b8a944da8c0b1f0, State: Initialized, Role: FOLLOWER
I20261116 21:09:32.371618 4706 consensus_queue.cc:261] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.371851 4706 ts_tablet_manager.cc:1480] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:32.371908 4299 heartbeater.cc:500] Master 127.0.117.254:33845 was elected leader, sending a full tablet report...
W20261116 21:09:32.376803 4300 tablet.cc:2441] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261116 21:09:32.406991 4562 tablet.cc:2441] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20261116 21:09:32.441808 4693 tablet.cc:2441] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20261116 21:09:32.577374 4714 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:32.577531 4714 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.577913 4714 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c619addfb98242838dc21a5efb20039d (127.0.117.196:40627), cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337)
I20261116 21:09:32.580818 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" is_pre_election: true
I20261116 21:09:32.581023 4515 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a67bb3cfd7a645629b8a944da8c0b1f0 in term 0.
I20261116 21:09:32.581255 4188 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a67bb3cfd7a645629b8a944da8c0b1f0, cd60873f44194f8d92ffc6a9fc4ac77c; no voters:
I20261116 21:09:32.581425 4714 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20261116 21:09:32.581478 4714 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:32.581504 4714 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:32.581614 4646 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c619addfb98242838dc21a5efb20039d" is_pre_election: true
I20261116 21:09:32.581768 4646 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a67bb3cfd7a645629b8a944da8c0b1f0 in term 0.
I20261116 21:09:32.582307 4714 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.582521 4714 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [CANDIDATE]: Term 1 election: Requested vote from peers c619addfb98242838dc21a5efb20039d (127.0.117.196:40627), cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337)
I20261116 21:09:32.582609 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
I20261116 21:09:32.582669 4515 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:32.582665 4646 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c619addfb98242838dc21a5efb20039d"
I20261116 21:09:32.582741 4646 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 0 FOLLOWER]: Advancing to term 1
I20261116 21:09:32.583525 4515 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67bb3cfd7a645629b8a944da8c0b1f0 in term 1.
I20261116 21:09:32.583653 4646 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a67bb3cfd7a645629b8a944da8c0b1f0 in term 1.
I20261116 21:09:32.583741 4188 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a67bb3cfd7a645629b8a944da8c0b1f0, cd60873f44194f8d92ffc6a9fc4ac77c; no voters:
I20261116 21:09:32.583863 4714 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 FOLLOWER]: Leader election won for term 1
I20261116 21:09:32.584069 4714 raft_consensus.cc:705] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 LEADER]: Becoming Leader. State: Replica: a67bb3cfd7a645629b8a944da8c0b1f0, State: Running, Role: LEADER
I20261116 21:09:32.584175 4714 consensus_queue.cc:238] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:32.585018 4111 catalog_manager.cc:5840] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193). New cstate: current_term: 1 leader_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } health_report { overall_health: HEALTHY } } }
I20261116 21:09:32.585184 4717 consensus_queue.cc:1049] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:32.585654 4717 consensus_queue.cc:1049] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20261116 21:09:32.589087 471 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20261116 21:09:32.589627 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a67bb3cfd7a645629b8a944da8c0b1f0 to finish bootstrapping
I20261116 21:09:32.593206 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cd60873f44194f8d92ffc6a9fc4ac77c to finish bootstrapping
I20261116 21:09:32.596313 471 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c619addfb98242838dc21a5efb20039d to finish bootstrapping
I20261116 21:09:32.599486 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4171
I20261116 21:09:32.605239 471 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0HHMU0/build/release/bin/kudu
/tmp/dist-test-task0HHMU0/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.117.193:39387
--local_ip_for_outbound_sockets=127.0.117.193
--tserver_master_addrs=127.0.117.254:33845
--webserver_port=33593
--webserver_interface=127.0.117.193
--builtin_ntp_servers=127.0.117.212:42421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000
--tablet_bootstrap_inject_latency_ms=3000 with env {}
W20261116 21:09:32.683171 4726 flags.cc:432] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20261116 21:09:32.683367 4726 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20261116 21:09:32.683393 4726 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20261116 21:09:32.683432 4726 flags.cc:432] Enabled unsafe flag: --tablet_bootstrap_inject_latency_ms=3000
W20261116 21:09:32.683455 4726 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20261116 21:09:32.685007 4726 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20261116 21:09:32.685081 4726 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20261116 21:09:32.685107 4726 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20261116 21:09:32.685145 4726 flags.cc:432] Enabled experimental flag: --log_target_replay_size_mb=1
W20261116 21:09:32.685194 4726 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.117.193
I20261116 21:09:32.686798 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.117.212:42421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--follower_unavailable_considered_failed_sec=3
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.117.193:39387
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.117.193
--webserver_port=33593
--tserver_master_addrs=127.0.117.254:33845
--tablet_bootstrap_inject_latency_ms=3000
--never_fsync=true
--heap_profile_path=/tmp/kudu.4726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.117.193
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3458f03c6ee765ff81fe1ff179b7d6d82b82420b
build type RELEASE
built by None at 16 Nov 2026 20:43:16 UTC on e7f111948823
build id 14816
I20261116 21:09:32.687055 4726 env_posix.cc:2314] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20261116 21:09:32.687238 4726 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20261116 21:09:32.689841 4726 server_base.cc:1061] running on GCE node
W20261116 21:09:32.689805 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.689811 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20261116 21:09:32.689811 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20261116 21:09:32.690131 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20261116 21:09:32.690325 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20261116 21:09:32.691453 4726 hybrid_clock.cc:648] HybridClock initialized: now 1794863372691443 us; error 29 us; skew 500 ppm
I20261116 21:09:32.692483 4726 webserver.cc:577] Webserver started at http://127.0.117.193:33593/ using document root <none> and password file <none>
I20261116 21:09:32.692737 4726 fs_manager.cc:362] Metadata directory not provided
I20261116 21:09:32.692794 4726 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20261116 21:09:32.694020 4726 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:32.694650 4740 log_block_manager.cc:4130] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20261116 21:09:32.694818 4726 fs_manager.cc:730] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20261116 21:09:32.694891 4726 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
format_stamp: "Formatted at 2026-11-16 21:09:31 on dist-test-slave-gg0q"
I20261116 21:09:32.695170 4726 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20261116 21:09:32.718736 4726 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20261116 21:09:32.719058 4726 env_posix.cc:2314] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20261116 21:09:32.719208 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20261116 21:09:32.719472 4726 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20261116 21:09:32.719954 4747 ts_tablet_manager.cc:588] Loading tablet metadata (0/1 complete)
I20261116 21:09:32.720806 4726 ts_tablet_manager.cc:631] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20261116 21:09:32.720867 4726 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20261116 21:09:32.720916 4726 ts_tablet_manager.cc:646] Registering tablets (0/1 complete)
I20261116 21:09:32.721571 4726 ts_tablet_manager.cc:662] Registered 1 tablets
W20261116 21:09:32.721614 4747 ts_tablet_manager.cc:1428] Injecting 3000ms delay in tablet bootstrapping
I20261116 21:09:32.721623 4726 ts_tablet_manager.cc:641] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20261116 21:09:32.727516 4726 rpc_server.cc:307] RPC server started. Bound to: 127.0.117.193:39387
I20261116 21:09:32.727553 4854 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.117.193:39387 every 8 connection(s)
I20261116 21:09:32.727958 4726 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20261116 21:09:32.729859 471 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0HHMU0/build/release/bin/kudu as pid 4726
I20261116 21:09:32.732786 4855 heartbeater.cc:345] Connected to a master server at 127.0.117.254:33845
I20261116 21:09:32.732910 4855 heartbeater.cc:462] Registering TS with master...
I20261116 21:09:32.733103 4855 heartbeater.cc:508] Master 127.0.117.254:33845 requested a full tablet report, sending...
I20261116 21:09:32.733649 4111 ts_manager.cc:210] Re-registered known tserver with Master: a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:32.734061 4111 master_service.cc:507] Signed X509 certificate for tserver {username='slave'} at 127.0.117.193:38651
I20261116 21:09:33.094066 4430 heartbeater.cc:500] Master 127.0.117.254:33845 was elected leader, sending a full tablet report...
I20261116 21:09:33.734980 4855 heartbeater.cc:500] Master 127.0.117.254:33845 was elected leader, sending a full tablet report...
I20261116 21:09:34.087514 4860 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a67bb3cfd7a645629b8a944da8c0b1f0)
I20261116 21:09:34.087656 4860 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:34.087728 4861 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a67bb3cfd7a645629b8a944da8c0b1f0)
I20261116 21:09:34.087843 4861 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:34.088094 4860 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c619addfb98242838dc21a5efb20039d (127.0.117.196:40627), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:34.088129 4861 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:34.092109 4646 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c619addfb98242838dc21a5efb20039d" is_pre_election: true
I20261116 21:09:34.092109 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" is_pre_election: true
I20261116 21:09:34.092227 4515 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c619addfb98242838dc21a5efb20039d in term 1.
I20261116 21:09:34.092227 4646 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd60873f44194f8d92ffc6a9fc4ac77c in term 1.
I20261116 21:09:34.092425 4581 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c619addfb98242838dc21a5efb20039d, cd60873f44194f8d92ffc6a9fc4ac77c; no voters:
I20261116 21:09:34.092542 4861 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 21:09:34.092525 4449 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c619addfb98242838dc21a5efb20039d, cd60873f44194f8d92ffc6a9fc4ac77c; no voters:
I20261116 21:09:34.092584 4861 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a67bb3cfd7a645629b8a944da8c0b1f0)
I20261116 21:09:34.092612 4861 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:34.092664 4860 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20261116 21:09:34.092710 4860 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Starting leader election (detected failure of leader a67bb3cfd7a645629b8a944da8c0b1f0)
I20261116 21:09:34.092729 4860 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 1 FOLLOWER]: Advancing to term 2
I20261116 21:09:34.093293 4861 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:34.093494 4861 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 election: Requested vote from peers cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:34.093525 4860 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:34.092669 4809 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" is_pre_election: true
I20261116 21:09:34.093678 4860 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 election: Requested vote from peers c619addfb98242838dc21a5efb20039d (127.0.117.196:40627), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:34.093775 4808 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" is_pre_election: true
I20261116 21:09:34.093820 4807 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
I20261116 21:09:34.093842 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
I20261116 21:09:34.093905 4806 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
W20261116 21:09:34.093900 4450 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
I20261116 21:09:34.093933 4515 raft_consensus.cc:2415] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c619addfb98242838dc21a5efb20039d in current term 2: Already voted for candidate cd60873f44194f8d92ffc6a9fc4ac77c in this term.
W20261116 21:09:34.094017 4450 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
W20261116 21:09:34.094168 4581 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
I20261116 21:09:34.094195 4646 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c619addfb98242838dc21a5efb20039d"
I20261116 21:09:34.094264 4646 raft_consensus.cc:2415] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cd60873f44194f8d92ffc6a9fc4ac77c in current term 2: Already voted for candidate c619addfb98242838dc21a5efb20039d in this term.
W20261116 21:09:34.094287 4581 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
I20261116 21:09:34.094352 4581 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c619addfb98242838dc21a5efb20039d; no voters: a67bb3cfd7a645629b8a944da8c0b1f0, cd60873f44194f8d92ffc6a9fc4ac77c
I20261116 21:09:34.094414 4861 raft_consensus.cc:2771] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261116 21:09:34.094409 4449 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd60873f44194f8d92ffc6a9fc4ac77c; no voters: a67bb3cfd7a645629b8a944da8c0b1f0, c619addfb98242838dc21a5efb20039d
I20261116 21:09:34.094515 4860 raft_consensus.cc:2771] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20261116 21:09:35.710479 4870 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20261116 21:09:35.710578 4870 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:35.710748 4870 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:35.711006 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" is_pre_election: true
I20261116 21:09:35.711127 4515 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c619addfb98242838dc21a5efb20039d in term 2.
I20261116 21:09:35.711092 4806 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" is_pre_election: true
I20261116 21:09:35.711340 4581 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c619addfb98242838dc21a5efb20039d, cd60873f44194f8d92ffc6a9fc4ac77c; no voters:
I20261116 21:09:35.711447 4870 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20261116 21:09:35.711452 4581 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
I20261116 21:09:35.711503 4870 raft_consensus.cc:499] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20261116 21:09:35.711524 4870 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 2 FOLLOWER]: Advancing to term 3
I20261116 21:09:35.712158 4870 raft_consensus.cc:521] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:35.712291 4870 leader_election.cc:290] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 election: Requested vote from peers cd60873f44194f8d92ffc6a9fc4ac77c (127.0.117.195:42337), a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387)
I20261116 21:09:35.712471 4515 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c"
I20261116 21:09:35.712484 4806 tablet_service.cc:1906] Received RequestConsensusVote() RPC: tablet_id: "4c4b46ee5b064b9fad4aded3b4fd1fc6" candidate_uuid: "c619addfb98242838dc21a5efb20039d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0"
I20261116 21:09:35.712550 4515 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 2 FOLLOWER]: Advancing to term 3
W20261116 21:09:35.712721 4581 leader_election.cc:343] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Illegal state: must be running to vote when last-logged opid is not known
I20261116 21:09:35.713181 4515 raft_consensus.cc:2490] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c619addfb98242838dc21a5efb20039d in term 3.
I20261116 21:09:35.713408 4581 leader_election.cc:304] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c619addfb98242838dc21a5efb20039d, cd60873f44194f8d92ffc6a9fc4ac77c; no voters: a67bb3cfd7a645629b8a944da8c0b1f0
I20261116 21:09:35.713528 4870 raft_consensus.cc:2826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 3 FOLLOWER]: Leader election won for term 3
I20261116 21:09:35.713697 4870 raft_consensus.cc:705] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [term 3 LEADER]: Becoming Leader. State: Replica: c619addfb98242838dc21a5efb20039d, State: Running, Role: LEADER
I20261116 21:09:35.713917 4870 consensus_queue.cc:238] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:35.714562 4515 raft_consensus.cc:1285] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c [term 3 FOLLOWER]: Refusing update from remote peer c619addfb98242838dc21a5efb20039d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20261116 21:09:35.714805 4111 catalog_manager.cc:5840] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d reported cstate change: term changed from 1 to 3, leader changed from a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193) to c619addfb98242838dc21a5efb20039d (127.0.117.196). New cstate: current_term: 3 leader_uuid: "c619addfb98242838dc21a5efb20039d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } health_report { overall_health: UNKNOWN } } }
I20261116 21:09:35.714975 4870 consensus_queue.cc:1049] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20261116 21:09:35.715128 4581 consensus_peers.cc:597] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d -> Peer a67bb3cfd7a645629b8a944da8c0b1f0 (127.0.117.193:39387): Couldn't send request to peer a67bb3cfd7a645629b8a944da8c0b1f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20261116 21:09:35.721873 4747 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Bootstrap starting.
I20261116 21:09:35.724853 4747 log.cc:826] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Log is configured to *not* fsync() on all Append() calls
I20261116 21:09:35.725960 4747 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20261116 21:09:35.726257 4747 tablet_bootstrap.cc:492] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Bootstrap complete.
I20261116 21:09:35.726512 4747 ts_tablet_manager.cc:1449] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20261116 21:09:35.727349 4747 raft_consensus.cc:360] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:35.727533 4747 raft_consensus.cc:748] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a67bb3cfd7a645629b8a944da8c0b1f0, State: Initialized, Role: FOLLOWER
I20261116 21:09:35.727655 4747 consensus_queue.cc:261] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c619addfb98242838dc21a5efb20039d" member_type: VOTER last_known_addr { host: "127.0.117.196" port: 40627 } } peers { permanent_uuid: "cd60873f44194f8d92ffc6a9fc4ac77c" member_type: VOTER last_known_addr { host: "127.0.117.195" port: 42337 } } peers { permanent_uuid: "a67bb3cfd7a645629b8a944da8c0b1f0" member_type: VOTER last_known_addr { host: "127.0.117.193" port: 39387 } }
I20261116 21:09:35.728670 4747 ts_tablet_manager.cc:1480] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20261116 21:09:36.293303 4806 raft_consensus.cc:3096] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0 [term 1 FOLLOWER]: Advancing to term 3
I20261116 21:09:36.367002 471 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20261116 21:09:36.367322 471 raft_config_change-itest.cc:610] Waiting until we've written at least 4MB...
I20261116 21:09:36.415272 4562 maintenance_manager.cc:419] P cd60873f44194f8d92ffc6a9fc4ac77c: Scheduling FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 3418722 bytes log retention, and flush 5079305 bytes memory
I20261116 21:09:36.421845 471 raft_config_change-itest.cc:616] Waiting for log GC on c619addfb98242838dc21a5efb20039d
I20261116 21:09:36.423804 4452 maintenance_manager.cc:643] P cd60873f44194f8d92ffc6a9fc4ac77c: FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.008s user 0.005s sys 0.000s Metrics: {"bytes_written":135937,"cfile_init":1,"compiler_manager_pool.queue_time_us":173,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":829,"drs_written":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4,"lbm_write_time_us":320,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":30,"thread_start_us":233,"threads_started":2}
I20261116 21:09:36.424050 4562 maintenance_manager.cc:419] P cd60873f44194f8d92ffc6a9fc4ac77c: Scheduling LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): free 3418722 bytes of WAL
I20261116 21:09:36.424224 4452 log_reader.cc:385] T 4c4b46ee5b064b9fad4aded3b4fd1fc6: removed 4 log segments from log reader
I20261116 21:09:36.424279 4452 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000001 (ops 1-9)
I20261116 21:09:36.424315 4452 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000002 (ops 10-16)
I20261116 21:09:36.424335 4452 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000003 (ops 17-21)
I20261116 21:09:36.424350 4452 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P cd60873f44194f8d92ffc6a9fc4ac77c: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000004 (ops 22-28)
I20261116 21:09:36.425251 4452 maintenance_manager.cc:643] P cd60873f44194f8d92ffc6a9fc4ac77c: LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:36.425449 4562 maintenance_manager.cc:419] P cd60873f44194f8d92ffc6a9fc4ac77c: Scheduling UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 470 bytes on disk
I20261116 21:09:36.425654 4452 maintenance_manager.cc:643] P cd60873f44194f8d92ffc6a9fc4ac77c: UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4}
I20261116 21:09:36.435052 4856 maintenance_manager.cc:419] P a67bb3cfd7a645629b8a944da8c0b1f0: Scheduling FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 4207719 bytes log retention, and flush 6119689 bytes memory
I20261116 21:09:36.442003 4745 maintenance_manager.cc:643] P a67bb3cfd7a645629b8a944da8c0b1f0: FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.007s user 0.001s sys 0.005s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":170,"dirs.queue_time_us":201,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":855,"drs_written":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4,"lbm_write_time_us":351,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":36,"thread_start_us":262,"threads_started":2}
I20261116 21:09:36.442212 4856 maintenance_manager.cc:419] P a67bb3cfd7a645629b8a944da8c0b1f0: Scheduling LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): free 4207719 bytes of WAL
I20261116 21:09:36.442361 4745 log_reader.cc:385] T 4c4b46ee5b064b9fad4aded3b4fd1fc6: removed 5 log segments from log reader
I20261116 21:09:36.442417 4745 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000001 (ops 1-9)
I20261116 21:09:36.442441 4745 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000002 (ops 10-16)
I20261116 21:09:36.442484 4745 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000003 (ops 17-22)
I20261116 21:09:36.442529 4745 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000004 (ops 23-29)
I20261116 21:09:36.442559 4745 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P a67bb3cfd7a645629b8a944da8c0b1f0: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000005 (ops 30-34)
I20261116 21:09:36.443876 4745 maintenance_manager.cc:643] P a67bb3cfd7a645629b8a944da8c0b1f0: LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:36.443991 4856 maintenance_manager.cc:419] P a67bb3cfd7a645629b8a944da8c0b1f0: Scheduling UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 535 bytes on disk
I20261116 21:09:36.444162 4745 maintenance_manager.cc:643] P a67bb3cfd7a645629b8a944da8c0b1f0: UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4}
I20261116 21:09:36.451208 4693 maintenance_manager.cc:419] P c619addfb98242838dc21a5efb20039d: Scheduling FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 4602274 bytes log retention, and flush 6119689 bytes memory
I20261116 21:09:36.456974 4583 maintenance_manager.cc:643] P c619addfb98242838dc21a5efb20039d: FlushMRSOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":506,"drs_written":1,"lbm_read_time_us":19,"lbm_reads_lt_1ms":4,"lbm_write_time_us":289,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":36,"thread_start_us":226,"threads_started":2}
I20261116 21:09:36.457280 4693 maintenance_manager.cc:419] P c619addfb98242838dc21a5efb20039d: Scheduling LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): free 4602274 bytes of WAL
I20261116 21:09:36.457436 4583 log_reader.cc:385] T 4c4b46ee5b064b9fad4aded3b4fd1fc6: removed 5 log segments from log reader
I20261116 21:09:36.457528 4583 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000001 (ops 1-9)
I20261116 21:09:36.457605 4583 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000002 (ops 10-16)
I20261116 21:09:36.457666 4583 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000003 (ops 17-23)
I20261116 21:09:36.457726 4583 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000004 (ops 24-30)
I20261116 21:09:36.457785 4583 log.cc:1079] T 4c4b46ee5b064b9fad4aded3b4fd1fc6 P c619addfb98242838dc21a5efb20039d: Deleting log segment in path: /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0/raft_consensus-itest-cluster/ts-3/wal/wals/4c4b46ee5b064b9fad4aded3b4fd1fc6/wal-000000005 (ops 31-37)
I20261116 21:09:36.459271 4583 maintenance_manager.cc:643] P c619addfb98242838dc21a5efb20039d: LogGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20261116 21:09:36.459427 4693 maintenance_manager.cc:419] P c619addfb98242838dc21a5efb20039d: Scheduling UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6): 535 bytes on disk
I20261116 21:09:36.459645 4583 maintenance_manager.cc:643] P c619addfb98242838dc21a5efb20039d: UndoDeltaBlockGCOp(4c4b46ee5b064b9fad4aded3b4fd1fc6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":23,"lbm_reads_lt_1ms":4}
I20261116 21:09:36.462728 471 raft_config_change-itest.cc:620] Log GC complete on c619addfb98242838dc21a5efb20039d
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_config_change-itest.cc:500: Failure
Value of: IsRaftConfigMember(added_replica_uuid, cstate.committed_config())
Actual: false
Expected: true
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20261116 21:10:06.555325 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4726
I20261116 21:10:06.561790 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4302
I20261116 21:10:06.566931 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4433
I20261116 21:10:06.572835 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4564
I20261116 21:10:06.578459 471 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0HHMU0/build/release/bin/kudu with pid 4080
2026-11-16T21:10:06Z chronyd exiting
I20261116 21:10:06.595341 471 test_util.cc:182] -----------------------------------------------
I20261116 21:10:06.595438 471 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-task0HHMU0/test-tmp/raft_config_change-itest.0.SlowTabletBootstrapTest.TestFallBehindSlowBootstrap.1794863344313111-471-0
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap (34894 ms)
[----------] 3 tests from SlowTabletBootstrapTest (49387 ms total)
[----------] Global test environment tear-down
[==========] 6 tests from 2 test suites ran. (62278 ms total)
[ PASSED ] 5 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SlowTabletBootstrapTest.TestFallBehindSlowBootstrap
1 FAILED TEST