Note: This is test shard 1 of 8.
[==========] Running 2 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from MaintenanceModeRF3ITest
[ RUN ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate
2025-06-27T13:55:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:55:16Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250627 13:55:16.094966 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:41403
--webserver_interface=127.8.167.254
--webserver_port=0
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.167.254:41403 with env {}
W20250627 13:55:16.415530 8871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:16.415860 8871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:16.415937 8871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:16.424085 8871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:55:16.424206 8871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:16.424264 8871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:55:16.424310 8871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:55:16.434314 8871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.167.254:41403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.167.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:16.435983 8871 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:16.437939 8871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:16.449493 8879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:16.449558 8877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:16.449867 8876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:16.450371 8871 server_base.cc:1048] running on GCE node
I20250627 13:55:16.451184 8871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:16.452790 8871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:16.454017 8871 hybrid_clock.cc:648] HybridClock initialized: now 1751032516453966 us; error 50 us; skew 500 ppm
I20250627 13:55:16.454434 8871 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:16.460052 8871 webserver.cc:469] Webserver started at http://127.8.167.254:35337/ using document root <none> and password file <none>
I20250627 13:55:16.460915 8871 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:16.461046 8871 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:16.461410 8871 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:55:16.464049 8871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/instance:
uuid: "93ef319ee202403688689c9c1fed179c"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:16.464908 8871 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal/instance:
uuid: "93ef319ee202403688689c9c1fed179c"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:16.470924 8871 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20250627 13:55:16.474735 8885 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:16.476205 8871 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250627 13:55:16.476395 8871 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
uuid: "93ef319ee202403688689c9c1fed179c"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:16.476622 8871 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:16.493103 8871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:16.494002 8871 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:16.494311 8871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:16.514320 8871 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.254:41403
I20250627 13:55:16.514328 8936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.254:41403 every 8 connection(s)
I20250627 13:55:16.515842 8871 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
I20250627 13:55:16.519837 8937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:16.524132 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 8871
I20250627 13:55:16.524403 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal/instance
I20250627 13:55:16.529578 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Bootstrap starting.
I20250627 13:55:16.533140 8937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:16.534246 8937 log.cc:826] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:55:16.536897 8937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: No bootstrap required, opened a new log
I20250627 13:55:16.543849 8937 raft_consensus.cc:357] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:16.544309 8937 raft_consensus.cc:383] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:16.544487 8937 raft_consensus.cc:738] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ef319ee202403688689c9c1fed179c, State: Initialized, Role: FOLLOWER
I20250627 13:55:16.545217 8937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:16.545486 8937 raft_consensus.cc:397] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:55:16.545605 8937 raft_consensus.cc:491] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:55:16.545773 8937 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:16.548305 8937 raft_consensus.cc:513] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:16.548911 8937 leader_election.cc:304] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ef319ee202403688689c9c1fed179c; no voters:
I20250627 13:55:16.549644 8937 leader_election.cc:290] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:55:16.549800 8942 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:16.551682 8937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:55:16.553893 8942 raft_consensus.cc:695] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 LEADER]: Becoming Leader. State: Replica: 93ef319ee202403688689c9c1fed179c, State: Running, Role: LEADER
I20250627 13:55:16.554579 8942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:16.557931 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ef319ee202403688689c9c1fed179c. Latest consensus state: current_term: 1 leader_uuid: "93ef319ee202403688689c9c1fed179c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } } }
I20250627 13:55:16.558385 8943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ef319ee202403688689c9c1fed179c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } } }
I20250627 13:55:16.559087 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: This master's current role is: LEADER
I20250627 13:55:16.559088 8943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: This master's current role is: LEADER
I20250627 13:55:16.561291 8956 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:55:16.566638 8956 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:55:16.575758 8956 catalog_manager.cc:1349] Generated new cluster ID: 399dc6bde91d44c9b89f31112cf9df43
I20250627 13:55:16.575912 8956 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:55:16.586556 8956 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:55:16.588265 8956 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:55:16.602121 8956 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Generated new TSK 0
I20250627 13:55:16.603253 8956 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:55:16.613400 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:0
--local_ip_for_outbound_sockets=127.8.167.193
--webserver_interface=127.8.167.193
--webserver_port=0
--tserver_master_addrs=127.8.167.254:41403
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
W20250627 13:55:16.945453 8961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:16.945792 8961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:16.945866 8961 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250627 13:55:16.945937 8961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:16.954137 8961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:16.954355 8961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:55:16.964095 8961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.8.167.254:41403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:16.965791 8961 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:16.967751 8961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:16.980101 8967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:16.980078 8969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:16.980185 8966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:16.981309 8961 server_base.cc:1048] running on GCE node
I20250627 13:55:16.981990 8961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:16.983062 8961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:16.984302 8961 hybrid_clock.cc:648] HybridClock initialized: now 1751032516984223 us; error 52 us; skew 500 ppm
I20250627 13:55:16.984602 8961 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:16.987212 8961 webserver.cc:469] Webserver started at http://127.8.167.193:34195/ using document root <none> and password file <none>
I20250627 13:55:16.988000 8961 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:16.988134 8961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:16.988497 8961 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:55:16.991173 8961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/instance:
uuid: "18c6bddc499b4363a95de5d2578e4a07"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:16.992151 8961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal/instance:
uuid: "18c6bddc499b4363a95de5d2578e4a07"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:16.998235 8961 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250627 13:55:17.002065 8975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.003554 8961 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:55:17.003746 8961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "18c6bddc499b4363a95de5d2578e4a07"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:17.003953 8961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:17.017079 8961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:17.017971 8961 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:17.018285 8961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:17.019279 8961 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:17.021006 8961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:55:17.021087 8961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.021220 8961 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:55:17.021279 8961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.055821 8961 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:45719
I20250627 13:55:17.055866 9087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:45719 every 8 connection(s)
I20250627 13:55:17.057268 8961 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:55:17.061972 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 8961
I20250627 13:55:17.062245 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal/instance
I20250627 13:55:17.065776 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.194:0
--local_ip_for_outbound_sockets=127.8.167.194
--webserver_interface=127.8.167.194
--webserver_port=0
--tserver_master_addrs=127.8.167.254:41403
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250627 13:55:17.072109 9088 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:17.072472 9088 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:17.073369 9088 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:17.075906 8902 ts_manager.cc:194] Registered new tserver with Master: 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719)
I20250627 13:55:17.078529 8902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:34273
W20250627 13:55:17.391546 9092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:17.391891 9092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:17.391976 9092 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250627 13:55:17.392043 9092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:17.400033 9092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:17.400244 9092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.194
I20250627 13:55:17.409798 9092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.167.194
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.8.167.254:41403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.194
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:17.411466 9092 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:17.413383 9092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:17.425096 9100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:17.425019 9097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:17.425056 9098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:17.425810 9092 server_base.cc:1048] running on GCE node
I20250627 13:55:17.426477 9092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:17.427425 9092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:17.428663 9092 hybrid_clock.cc:648] HybridClock initialized: now 1751032517428603 us; error 37 us; skew 500 ppm
I20250627 13:55:17.428952 9092 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:17.431491 9092 webserver.cc:469] Webserver started at http://127.8.167.194:44951/ using document root <none> and password file <none>
I20250627 13:55:17.432274 9092 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:17.432412 9092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:17.432772 9092 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:55:17.435318 9092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data/instance:
uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.436108 9092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal/instance:
uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.441601 9092 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20250627 13:55:17.445155 9106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.446537 9092 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:55:17.446702 9092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal
uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.446882 9092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:17.474156 9092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:17.475054 9092 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:17.475345 9092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:17.476357 9092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:17.478068 9092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:55:17.478153 9092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250627 13:55:17.478294 9092 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:55:17.478351 9092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.512085 9092 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.194:35625
I20250627 13:55:17.512133 9218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.194:35625 every 8 connection(s)
I20250627 13:55:17.513531 9092 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
I20250627 13:55:17.514900 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 9092
I20250627 13:55:17.515085 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-1/wal/instance
I20250627 13:55:17.519227 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.195:0
--local_ip_for_outbound_sockets=127.8.167.195
--webserver_interface=127.8.167.195
--webserver_port=0
--tserver_master_addrs=127.8.167.254:41403
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250627 13:55:17.531426 9219 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:17.531847 9219 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:17.532992 9219 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:17.534960 8902 ts_manager.cc:194] Registered new tserver with Master: e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:17.536281 8902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.194:39379
W20250627 13:55:17.845108 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:17.845455 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:17.845530 9222 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250627 13:55:17.845597 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:17.853675 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:17.853885 9222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.195
I20250627 13:55:17.863411 9222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.167.195
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.8.167.254:41403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.195
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:17.865029 9222 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:17.866887 9222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:17.879127 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:17.879040 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:17.879163 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:17.879904 9222 server_base.cc:1048] running on GCE node
I20250627 13:55:17.880494 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:17.881397 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:17.882611 9222 hybrid_clock.cc:648] HybridClock initialized: now 1751032517882548 us; error 42 us; skew 500 ppm
I20250627 13:55:17.882899 9222 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:17.885215 9222 webserver.cc:469] Webserver started at http://127.8.167.195:33633/ using document root <none> and password file <none>
I20250627 13:55:17.885939 9222 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:17.886080 9222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:17.886412 9222 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:55:17.888895 9222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data/instance:
uuid: "8064817311e242618056800a0ca537dd"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.889674 9222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal/instance:
uuid: "8064817311e242618056800a0ca537dd"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.894958 9222 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250627 13:55:17.898303 9236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.899546 9222 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250627 13:55:17.899703 9222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal
uuid: "8064817311e242618056800a0ca537dd"
format_stamp: "Formatted at 2025-06-27 13:55:17 on dist-test-slave-1xs4"
I20250627 13:55:17.899868 9222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:17.924379 9222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:17.925284 9222 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:17.925585 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:17.926609 9222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:17.928396 9222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:55:17.928486 9222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.928627 9222 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:55:17.928686 9222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:17.964638 9222 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.195:43925
I20250627 13:55:17.964691 9348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.195:43925 every 8 connection(s)
I20250627 13:55:17.966094 9222 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
I20250627 13:55:17.969172 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 9222
I20250627 13:55:17.969362 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-2/wal/instance
I20250627 13:55:17.979549 9349 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:17.979900 9349 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:17.980763 9349 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:17.982440 8902 ts_manager.cc:194] Registered new tserver with Master: 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:17.983246 8902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.195:48481
I20250627 13:55:17.985939 8863 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 13:55:18.004530 8863 test_util.cc:276] Using random seed: 121229064
I20250627 13:55:18.026665 8902 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38770:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000UUU\025\004\001\000\252\252\252*\004\001\000\377\377\377?\004\001\000TUUU\004\001\000\251\252\252j"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250627 13:55:18.028578 8902 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 13:55:18.082370 9088 heartbeater.cc:499] Master 127.8.167.254:41403 was elected leader, sending a full tablet report...
I20250627 13:55:18.084395 9149 tablet_service.cc:1468] Processing CreateTablet for tablet ab269b1c09ed41b0b51072f18c2462d8 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250627 13:55:18.084856 9284 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef29c61340f4869b8703e67f2e32f14 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250627 13:55:18.086557 9149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab269b1c09ed41b0b51072f18c2462d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.087864 9151 tablet_service.cc:1468] Processing CreateTablet for tablet c370dad30118471cbdc7094f7209aad5 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250627 13:55:18.088371 9151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c370dad30118471cbdc7094f7209aad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.087567 9022 tablet_service.cc:1468] Processing CreateTablet for tablet 22aa1c36043f4e8781bac641bd1035aa (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250627 13:55:18.088799 9018 tablet_service.cc:1468] Processing CreateTablet for tablet ab269b1c09ed41b0b51072f18c2462d8 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250627 13:55:18.089329 9019 tablet_service.cc:1468] Processing CreateTablet for tablet 82aebc7eff8c4c8a86eb994bf24bd99c (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250627 13:55:18.089517 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aa1c36043f4e8781bac641bd1035aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.090056 9021 tablet_service.cc:1468] Processing CreateTablet for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250627 13:55:18.090190 9153 tablet_service.cc:1468] Processing CreateTablet for tablet 22aa1c36043f4e8781bac641bd1035aa (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250627 13:55:18.090456 9018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab269b1c09ed41b0b51072f18c2462d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.090617 9153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aa1c36043f4e8781bac641bd1035aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.087708 9023 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef29c61340f4869b8703e67f2e32f14 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250627 13:55:18.090972 9023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef29c61340f4869b8703e67f2e32f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.084514 9154 tablet_service.cc:1468] Processing CreateTablet for tablet 9ef29c61340f4869b8703e67f2e32f14 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION VALUES < 357913941
I20250627 13:55:18.091534 9154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef29c61340f4869b8703e67f2e32f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.091562 9020 tablet_service.cc:1468] Processing CreateTablet for tablet c370dad30118471cbdc7094f7209aad5 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250627 13:55:18.092003 9020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c370dad30118471cbdc7094f7209aad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.092182 9150 tablet_service.cc:1468] Processing CreateTablet for tablet 82aebc7eff8c4c8a86eb994bf24bd99c (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250627 13:55:18.092602 9019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aebc7eff8c4c8a86eb994bf24bd99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.092612 9150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aebc7eff8c4c8a86eb994bf24bd99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.092130 9152 tablet_service.cc:1468] Processing CreateTablet for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250627 13:55:18.093045 9282 tablet_service.cc:1468] Processing CreateTablet for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1073741823
I20250627 13:55:18.093501 9152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55dc8052c494b8d8a79c2e7f9ed39d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.093871 9282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55dc8052c494b8d8a79c2e7f9ed39d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.095305 9281 tablet_service.cc:1468] Processing CreateTablet for tablet c370dad30118471cbdc7094f7209aad5 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1073741823 <= VALUES < 1431655764
I20250627 13:55:18.095810 9281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c370dad30118471cbdc7094f7209aad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.096741 9279 tablet_service.cc:1468] Processing CreateTablet for tablet ab269b1c09ed41b0b51072f18c2462d8 (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1789569705 <= VALUES
I20250627 13:55:18.097206 9279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab269b1c09ed41b0b51072f18c2462d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.098096 9021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55dc8052c494b8d8a79c2e7f9ed39d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.102582 9280 tablet_service.cc:1468] Processing CreateTablet for tablet 82aebc7eff8c4c8a86eb994bf24bd99c (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1789569705
I20250627 13:55:18.103107 9280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aebc7eff8c4c8a86eb994bf24bd99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.118180 9283 tablet_service.cc:1468] Processing CreateTablet for tablet 22aa1c36043f4e8781bac641bd1035aa (DEFAULT_TABLE table=test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]), partition=RANGE (key) PARTITION 357913941 <= VALUES < 715827882
I20250627 13:55:18.118832 9283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aa1c36043f4e8781bac641bd1035aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.122335 9284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef29c61340f4869b8703e67f2e32f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:18.124645 9374 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
I20250627 13:55:18.132139 9374 tablet_bootstrap.cc:654] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.133423 9374 log.cc:826] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: Log is configured to *not* fsync() on all Append() calls
I20250627 13:55:18.139922 9374 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.140470 9374 ts_tablet_manager.cc:1397] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20250627 13:55:18.146771 9373 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.150213 9373 tablet_bootstrap.cc:654] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.150961 9372 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.151388 9374 raft_consensus.cc:357] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.151693 9373 log.cc:826] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Log is configured to *not* fsync() on all Append() calls
I20250627 13:55:18.151759 9374 raft_consensus.cc:383] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.151887 9374 raft_consensus.cc:738] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.152868 9374 consensus_queue.cc:260] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.153324 9376 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.153635 9376 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.155483 9372 tablet_bootstrap.cc:654] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.156098 9376 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.156780 9219 heartbeater.cc:499] Master 127.8.167.254:41403 was elected leader, sending a full tablet report...
I20250627 13:55:18.156941 9372 log.cc:826] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Log is configured to *not* fsync() on all Append() calls
I20250627 13:55:18.164119 9373 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.164530 9373 ts_tablet_manager.cc:1397] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20250627 13:55:18.165077 9374 ts_tablet_manager.cc:1428] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.024s user 0.022s sys 0.000s
I20250627 13:55:18.165652 9374 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
I20250627 13:55:18.167882 9374 tablet_bootstrap.cc:654] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.178747 9374 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.178972 9374 ts_tablet_manager.cc:1397] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20250627 13:55:18.178846 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.179862 9374 raft_consensus.cc:357] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.180042 9374 raft_consensus.cc:383] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.180115 9374 raft_consensus.cc:738] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.180485 9374 consensus_queue.cc:260] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.180986 9374 ts_tablet_manager.cc:1428] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250627 13:55:18.181560 9109 leader_election.cc:343] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.181684 9374 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
I20250627 13:55:18.182451 9372 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.182780 9372 ts_tablet_manager.cc:1397] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.008s
I20250627 13:55:18.183619 9373 raft_consensus.cc:357] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.183904 9374 tablet_bootstrap.cc:654] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.183982 9373 raft_consensus.cc:383] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.184099 9373 raft_consensus.cc:738] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.184912 9373 consensus_queue.cc:260] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.190927 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.191874 9372 raft_consensus.cc:357] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.192269 9372 raft_consensus.cc:383] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.192324 9374 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.192417 9372 raft_consensus.cc:738] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
W20250627 13:55:18.192329 9109 leader_election.cc:343] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.192631 9374 ts_tablet_manager.cc:1397] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250627 13:55:18.192634 9109 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd
I20250627 13:55:18.193238 9372 consensus_queue.cc:260] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.193678 9374 raft_consensus.cc:357] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.193892 9374 raft_consensus.cc:383] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.194008 9374 raft_consensus.cc:738] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.194259 9373 ts_tablet_manager.cc:1428] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.030s user 0.011s sys 0.009s
I20250627 13:55:18.194265 9378 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:55:18.194308 9374 consensus_queue.cc:260] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.194945 9374 ts_tablet_manager.cc:1428] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:55:18.195328 9374 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
I20250627 13:55:18.196192 9373 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.197707 9374 tablet_bootstrap.cc:654] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.198148 9373 tablet_bootstrap.cc:654] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.201573 9372 ts_tablet_manager.cc:1428] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
I20250627 13:55:18.201959 9349 heartbeater.cc:499] Master 127.8.167.254:41403 was elected leader, sending a full tablet report...
I20250627 13:55:18.205592 9372 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.205937 9378 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.206169 9378 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.207108 9378 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.207814 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.207892 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.208386 9372 tablet_bootstrap.cc:654] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
W20250627 13:55:18.208410 9109 leader_election.cc:343] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Illegal state: must be running to vote when last-logged opid is not known
W20250627 13:55:18.208707 9109 leader_election.cc:343] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.208824 9109 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd
I20250627 13:55:18.209244 9378 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:55:18.210175 9373 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.210378 9373 ts_tablet_manager.cc:1397] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20250627 13:55:18.211031 9374 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.211298 9374 ts_tablet_manager.cc:1397] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20250627 13:55:18.211472 9373 raft_consensus.cc:357] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.211699 9373 raft_consensus.cc:383] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.211802 9373 raft_consensus.cc:738] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.212074 9373 consensus_queue.cc:260] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.212303 9374 raft_consensus.cc:357] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.212538 9374 raft_consensus.cc:383] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.212617 9373 ts_tablet_manager.cc:1428] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250627 13:55:18.212627 9374 raft_consensus.cc:738] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.212988 9373 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.213002 9374 consensus_queue.cc:260] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.213167 9372 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.213193 9378 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.213395 9372 ts_tablet_manager.cc:1397] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250627 13:55:18.213335 9378 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.213964 9378 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.214497 9372 raft_consensus.cc:357] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.214679 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.214733 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.214756 9372 raft_consensus.cc:383] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.214869 9373 tablet_bootstrap.cc:654] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.214936 9372 raft_consensus.cc:738] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.215207 9372 consensus_queue.cc:260] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
W20250627 13:55:18.215301 9109 leader_election.cc:343] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925): Illegal state: must be running to vote when last-logged opid is not known
W20250627 13:55:18.215523 9109 leader_election.cc:343] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.215667 9109 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd
I20250627 13:55:18.215898 9372 ts_tablet_manager.cc:1428] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250627 13:55:18.216092 9378 raft_consensus.cc:2747] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:55:18.216341 9372 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.216688 9374 ts_tablet_manager.cc:1428] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:55:18.217003 9374 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
W20250627 13:55:18.218061 9350 tablet.cc:2378] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:55:18.218259 9372 tablet_bootstrap.cc:654] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.218801 9374 tablet_bootstrap.cc:654] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.226684 9384 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.226964 9384 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.227213 9372 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.227360 9372 ts_tablet_manager.cc:1397] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20250627 13:55:18.227988 9372 raft_consensus.cc:357] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.228127 9372 raft_consensus.cc:383] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.228184 9372 raft_consensus.cc:738] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.228344 9372 consensus_queue.cc:260] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.228821 9372 ts_tablet_manager.cc:1428] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250627 13:55:18.228821 9384 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.229151 9372 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.230396 9373 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.230564 9373 ts_tablet_manager.cc:1397] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20250627 13:55:18.231060 9372 tablet_bootstrap.cc:654] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.231490 9373 raft_consensus.cc:357] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.231698 9373 raft_consensus.cc:383] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.231796 9373 raft_consensus.cc:738] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.232198 9373 consensus_queue.cc:260] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.232924 9373 ts_tablet_manager.cc:1428] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250627 13:55:18.233938 9373 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.236516 9373 tablet_bootstrap.cc:654] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.242465 9374 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.242749 9374 ts_tablet_manager.cc:1397] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20250627 13:55:18.243749 9374 raft_consensus.cc:357] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.244050 9374 raft_consensus.cc:383] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.244226 9374 raft_consensus.cc:738] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.244680 9374 consensus_queue.cc:260] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.246272 9374 ts_tablet_manager.cc:1428] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:55:18.246634 9374 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: Bootstrap starting.
I20250627 13:55:18.248625 9374 tablet_bootstrap.cc:654] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.250690 9378 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.250900 9378 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.251557 9378 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.252693 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.252976 9043 raft_consensus.cc:2466] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 0.
I20250627 13:55:18.253430 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
W20250627 13:55:18.254365 9109 leader_election.cc:343] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.254663 9109 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 8064817311e242618056800a0ca537dd
I20250627 13:55:18.255180 9378 raft_consensus.cc:2802] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.255295 9378 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.255407 9378 raft_consensus.cc:3058] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.258960 9378 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.259023 9385 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.259402 9385 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.260406 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07"
I20250627 13:55:18.260679 9043 raft_consensus.cc:3058] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.265306 9043 raft_consensus.cc:2466] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 1.
W20250627 13:55:18.265766 9220 tablet.cc:2378] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:55:18.265810 9376 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.266070 9376 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.266758 9376 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.267295 9378 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.269052 9374 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: No bootstrap required, opened a new log
I20250627 13:55:18.269304 9374 ts_tablet_manager.cc:1397] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250627 13:55:18.270179 9109 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.270555 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.270841 9304 raft_consensus.cc:2466] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 0.
I20250627 13:55:18.271248 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd"
I20250627 13:55:18.271735 9378 raft_consensus.cc:2802] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.271915 9378 raft_consensus.cc:695] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Becoming Leader. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Running, Role: LEADER
I20250627 13:55:18.272466 9378 consensus_queue.cc:237] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.261991 9385 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.273396 9385 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.273600 9385 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.274377 9042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.274653 9042 raft_consensus.cc:2466] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 0.
I20250627 13:55:18.274951 9041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.275310 9041 raft_consensus.cc:2466] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 0.
I20250627 13:55:18.275628 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.275847 9043 raft_consensus.cc:2466] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 0.
I20250627 13:55:18.276180 9239 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd; no voters:
I20250627 13:55:18.276556 9109 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.277081 9385 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
W20250627 13:55:18.277169 9109 leader_election.cc:343] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.277593 9376 raft_consensus.cc:2802] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.277124 9398 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.277706 9376 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.277830 9376 raft_consensus.cc:3058] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.277849 9239 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd; no voters:
I20250627 13:55:18.277917 9385 raft_consensus.cc:2802] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.278169 9385 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.277829 9398 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.279829 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.278764 9385 raft_consensus.cc:3058] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
W20250627 13:55:18.280468 9239 leader_election.cc:343] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:55:18.281217 9376 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.281746 9372 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.281929 9372 ts_tablet_manager.cc:1397] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.053s user 0.017s sys 0.030s
I20250627 13:55:18.282768 9372 raft_consensus.cc:357] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.282991 9372 raft_consensus.cc:383] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.283085 9372 raft_consensus.cc:738] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.283335 9372 consensus_queue.cc:260] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.284387 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07"
I20250627 13:55:18.284467 9385 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.284660 9043 raft_consensus.cc:3058] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.285952 9041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07"
I20250627 13:55:18.286230 9041 raft_consensus.cc:3058] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.288800 9043 raft_consensus.cc:2466] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 1.
I20250627 13:55:18.289796 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd"
I20250627 13:55:18.290122 9303 raft_consensus.cc:3058] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.290573 9041 raft_consensus.cc:2466] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 1.
I20250627 13:55:18.291213 9239 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd; no voters:
I20250627 13:55:18.295548 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.296241 9174 raft_consensus.cc:2391] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18c6bddc499b4363a95de5d2578e4a07 in current term 1: Already voted for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in this term.
I20250627 13:55:18.280929 9398 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.296998 9109 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.297487 9376 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.297770 9376 raft_consensus.cc:2802] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.297962 9385 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.298152 9376 raft_consensus.cc:695] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Becoming Leader. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Running, Role: LEADER
I20250627 13:55:18.298213 9373 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.298394 9373 ts_tablet_manager.cc:1397] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.017s
I20250627 13:55:18.298580 9376 consensus_queue.cc:237] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.299238 9373 raft_consensus.cc:357] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.299458 9373 raft_consensus.cc:383] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.299558 9373 raft_consensus.cc:738] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.299789 9373 consensus_queue.cc:260] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.300256 9385 raft_consensus.cc:2802] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.300459 9385 raft_consensus.cc:695] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 1 LEADER]: Becoming Leader. State: Replica: 8064817311e242618056800a0ca537dd, State: Running, Role: LEADER
I20250627 13:55:18.300943 9385 consensus_queue.cc:237] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.301501 9303 raft_consensus.cc:2466] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 1.
I20250627 13:55:18.302254 9398 raft_consensus.cc:2762] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250627 13:55:18.303247 9373 ts_tablet_manager.cc:1428] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250627 13:55:18.303108 9374 raft_consensus.cc:357] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.304172 9374 raft_consensus.cc:383] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.304487 9374 raft_consensus.cc:738] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.304919 9374 consensus_queue.cc:260] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.305836 9374 ts_tablet_manager.cc:1428] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20250627 13:55:18.306452 9373 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.310509 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.310942 9373 tablet_bootstrap.cc:654] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.310964 9303 raft_consensus.cc:2391] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18c6bddc499b4363a95de5d2578e4a07 in current term 1: Already voted for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in this term.
W20250627 13:55:18.311022 9089 tablet.cc:2378] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:55:18.311775 8978 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:18.312213 9384 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:55:18.319362 9373 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.319559 9373 ts_tablet_manager.cc:1397] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20250627 13:55:18.320219 9373 raft_consensus.cc:357] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.320353 9373 raft_consensus.cc:383] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.320413 9373 raft_consensus.cc:738] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.320618 9373 consensus_queue.cc:260] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.320250 8901 catalog_manager.cc:5582] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194). New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:18.321552 9373 ts_tablet_manager.cc:1428] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:55:18.321830 9373 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:18.323323 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.323535 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
I20250627 13:55:18.323734 9172 raft_consensus.cc:2466] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 0.
I20250627 13:55:18.323782 9171 raft_consensus.cc:3058] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.323345 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.325040 9373 tablet_bootstrap.cc:654] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.327064 9372 ts_tablet_manager.cc:1428] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.045s user 0.012s sys 0.028s
I20250627 13:55:18.327378 9171 raft_consensus.cc:2466] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 1.
I20250627 13:55:18.327531 9372 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.323794 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.327755 9174 raft_consensus.cc:2374] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8064817311e242618056800a0ca537dd in term 1. Re-sending same reply.
I20250627 13:55:18.328298 9240 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07
I20250627 13:55:18.329473 9372 tablet_bootstrap.cc:654] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.330479 9399 raft_consensus.cc:2802] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.330585 9399 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.330714 9399 raft_consensus.cc:3058] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.331209 8900 catalog_manager.cc:5582] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194). New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:18.333122 9397 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.333304 9397 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.334578 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.334684 9399 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.334905 9043 raft_consensus.cc:2391] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8064817311e242618056800a0ca537dd in current term 1: Already voted for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in this term.
I20250627 13:55:18.334869 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.335417 9239 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8064817311e242618056800a0ca537dd; no voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:18.335559 9397 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.335743 9397 raft_consensus.cc:3058] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.335852 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07"
I20250627 13:55:18.336010 9043 raft_consensus.cc:3058] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.337039 9399 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.337070 9373 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: No bootstrap required, opened a new log
I20250627 13:55:18.337136 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
I20250627 13:55:18.337265 9373 ts_tablet_manager.cc:1397] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20250627 13:55:18.337361 9174 raft_consensus.cc:3058] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.338124 9373 raft_consensus.cc:357] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.338322 9373 raft_consensus.cc:383] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.338390 9373 raft_consensus.cc:738] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.338670 9373 consensus_queue.cc:260] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.339027 9043 raft_consensus.cc:2466] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 1.
I20250627 13:55:18.339469 9239 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd; no voters:
I20250627 13:55:18.339881 9372 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.340029 9373 ts_tablet_manager.cc:1428] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:55:18.340035 9372 ts_tablet_manager.cc:1397] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250627 13:55:18.340549 9399 raft_consensus.cc:2802] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.340720 9174 raft_consensus.cc:2466] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 1.
I20250627 13:55:18.340809 9399 raft_consensus.cc:695] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 1 LEADER]: Becoming Leader. State: Replica: 8064817311e242618056800a0ca537dd, State: Running, Role: LEADER
I20250627 13:55:18.340924 9372 raft_consensus.cc:357] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.341145 9372 raft_consensus.cc:383] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.341121 9399 consensus_queue.cc:237] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.341274 9372 raft_consensus.cc:738] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.341645 9372 consensus_queue.cc:260] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.342218 9372 ts_tablet_manager.cc:1428] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250627 13:55:18.342525 9372 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: Bootstrap starting.
I20250627 13:55:18.343858 9397 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250627 13:55:18.344551 9372 tablet_bootstrap.cc:654] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: Neither blocks nor log segments found. Creating new log.
I20250627 13:55:18.351810 9372 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: No bootstrap required, opened a new log
I20250627 13:55:18.352010 9372 ts_tablet_manager.cc:1397] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250627 13:55:18.352519 9378 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.352702 9378 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.352769 9372 raft_consensus.cc:357] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.352979 9372 raft_consensus.cc:383] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:55:18.353066 9372 raft_consensus.cc:738] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8064817311e242618056800a0ca537dd, State: Initialized, Role: FOLLOWER
I20250627 13:55:18.353369 9372 consensus_queue.cc:260] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.353463 9378 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.353461 8901 catalog_manager.cc:5582] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8064817311e242618056800a0ca537dd (127.8.167.195). New cstate: current_term: 1 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } }
I20250627 13:55:18.353871 9372 ts_tablet_manager.cc:1428] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250627 13:55:18.353971 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07" is_pre_election: true
I20250627 13:55:18.354249 9043 raft_consensus.cc:2466] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 0.
I20250627 13:55:18.354513 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.354666 9109 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.354705 9303 raft_consensus.cc:2466] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 0.
I20250627 13:55:18.354929 9378 raft_consensus.cc:2802] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.355048 9378 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.355134 9378 raft_consensus.cc:3058] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.357409 9378 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.358014 9378 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:18.358453 9043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6bddc499b4363a95de5d2578e4a07"
I20250627 13:55:18.358551 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd"
I20250627 13:55:18.358669 9043 raft_consensus.cc:3058] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.358773 9303 raft_consensus.cc:3058] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.359067 8901 catalog_manager.cc:5582] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8064817311e242618056800a0ca537dd (127.8.167.195). New cstate: current_term: 1 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } }
I20250627 13:55:18.360978 9043 raft_consensus.cc:2466] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 1.
I20250627 13:55:18.361125 9303 raft_consensus.cc:2466] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e07fa0ee9b1b4e28b800d1ac6da68e17 in term 1.
I20250627 13:55:18.361421 9109 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.361747 9378 raft_consensus.cc:2802] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.361933 9378 raft_consensus.cc:695] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Becoming Leader. State: Replica: e07fa0ee9b1b4e28b800d1ac6da68e17, State: Running, Role: LEADER
I20250627 13:55:18.362298 9378 consensus_queue.cc:237] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:18.365027 8901 catalog_manager.cc:5582] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194). New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:18.365824 9384 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:18.365973 9384 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.366715 9384 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.367187 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:18.367163 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:18.367422 9174 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18c6bddc499b4363a95de5d2578e4a07 in term 0.
I20250627 13:55:18.367441 9303 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18c6bddc499b4363a95de5d2578e4a07 in term 0.
I20250627 13:55:18.367878 8979 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.368153 9384 raft_consensus.cc:2802] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:55:18.368291 9384 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:55:18.368400 9384 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.370625 9384 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.371176 9384 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 election: Requested vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:18.371611 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd"
I20250627 13:55:18.371660 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
I20250627 13:55:18.371812 9303 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.371836 9174 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:55:18.373819 9303 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c6bddc499b4363a95de5d2578e4a07 in term 1.
I20250627 13:55:18.373833 9174 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c6bddc499b4363a95de5d2578e4a07 in term 1.
I20250627 13:55:18.374373 8978 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:18.374749 9384 raft_consensus.cc:2802] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:55:18.375208 9384 raft_consensus.cc:695] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 LEADER]: Becoming Leader. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Running, Role: LEADER
I20250627 13:55:18.375721 9384 consensus_queue.cc:237] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:18.380903 8901 catalog_manager.cc:5582] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193). New cstate: current_term: 1 leader_uuid: "18c6bddc499b4363a95de5d2578e4a07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:18.385834 9376 consensus_queue.cc:1035] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.387233 9376 consensus_queue.cc:1035] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.394379 9402 consensus_queue.cc:1035] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.401178 9376 consensus_queue.cc:1035] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.415336 9399 consensus_queue.cc:1035] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.417824 9399 consensus_queue.cc:1035] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.433514 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.196:0
--local_ip_for_outbound_sockets=127.8.167.196
--webserver_interface=127.8.167.196
--webserver_port=0
--tserver_master_addrs=127.8.167.254:41403
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
I20250627 13:55:18.450608 9376 consensus_queue.cc:1035] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.458657 9400 consensus_queue.cc:1035] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.461772 9400 consensus_queue.cc:1035] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.471990 9376 consensus_queue.cc:1035] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.480623 9411 consensus_queue.cc:1035] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:55:18.482580 9384 consensus_queue.cc:1035] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250627 13:55:19.229940 9426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:19.230422 9426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:19.230551 9426 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250627 13:55:19.230648 9426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:19.242937 9426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:19.243227 9426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.196
I20250627 13:55:19.258498 9426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.8.167.196
--webserver_port=0
--enable_log_gc=false
--tserver_master_addrs=127.8.167.254:41403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.196
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:19.260802 9426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:19.263481 9426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:19.433420 9466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:19.436091 9465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:19.544217 9426 server_base.cc:1048] running on GCE node
W20250627 13:55:19.567566 9468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:19.568751 9426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:19.678905 9426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:19.680831 9426 hybrid_clock.cc:648] HybridClock initialized: now 1751032519680743 us; error 53 us; skew 500 ppm
I20250627 13:55:19.681393 9426 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:19.714525 9426 webserver.cc:469] Webserver started at http://127.8.167.196:43649/ using document root <none> and password file <none>
I20250627 13:55:19.715857 9426 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:19.716143 9426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:19.716710 9426 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:55:19.720381 9426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data/instance:
uuid: "5fe0be3d9258445cbdc65ddc3a59f144"
format_stamp: "Formatted at 2025-06-27 13:55:19 on dist-test-slave-1xs4"
I20250627 13:55:19.721755 9426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal/instance:
uuid: "5fe0be3d9258445cbdc65ddc3a59f144"
format_stamp: "Formatted at 2025-06-27 13:55:19 on dist-test-slave-1xs4"
I20250627 13:55:19.752375 9426 fs_manager.cc:696] Time spent creating directory manager: real 0.030s user 0.012s sys 0.012s
I20250627 13:55:19.774121 9474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:19.776352 9426 fs_manager.cc:730] Time spent opening block manager: real 0.019s user 0.011s sys 0.002s
I20250627 13:55:19.776553 9426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal
uuid: "5fe0be3d9258445cbdc65ddc3a59f144"
format_stamp: "Formatted at 2025-06-27 13:55:19 on dist-test-slave-1xs4"
I20250627 13:55:19.776823 9426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:19.932754 9426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:19.934157 9426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:19.934492 9426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:19.935771 9426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:19.955157 9426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:55:19.955286 9426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:19.955416 9426 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:55:19.955471 9426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:21.400600 9598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.196:34697 every 8 connection(s)
I20250627 13:55:21.400957 9426 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.196:34697
I20250627 13:55:21.403038 9426 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
I20250627 13:55:21.407989 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 9426
I20250627 13:55:21.408195 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-3/wal/instance
I20250627 13:55:21.517526 9599 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:21.518070 9599 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:21.530620 9599 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:21.543852 8897 ts_manager.cc:194] Registered new tserver with Master: 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697)
I20250627 13:55:21.545271 8897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.196:50781
I20250627 13:55:22.397580 8897 ts_manager.cc:295] Set tserver state for 18c6bddc499b4363a95de5d2578e4a07 to MAINTENANCE_MODE
I20250627 13:55:22.398751 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 8961
I20250627 13:55:22.400167 9083 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:55:22.552533 9599 heartbeater.cc:499] Master 127.8.167.254:41403 was elected leader, sending a full tablet report...
I20250627 13:55:22.650190 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 8961
W20250627 13:55:22.689055 9359 connection.cc:537] client connection to 127.8.167.193:45719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.689450 9359 meta_cache.cc:302] tablet 9ef29c61340f4869b8703e67f2e32f14: replica 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.689448 9109 connection.cc:537] client connection to 127.8.167.193:45719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.689636 9240 connection.cc:537] server connection from 127.8.167.193:54083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.689755 9239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.167.193:45719 (error 108)
W20250627 13:55:22.689821 9109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.690874 9109 connection.cc:537] server connection from 127.8.167.193:44281 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:22.692394 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:22.692555 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:22.692662 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:22.692793 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:22.692903 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:22.695060 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.695015 9259 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.699235 9134 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.699299 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.715545 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.719231 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.723976 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.730232 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.744606 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.755357 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.756826 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.769985 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.785446 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.797528 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.799259 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.811025 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.831661 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.847671 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.849732 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.866832 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.888494 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.909651 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.909672 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
W20250627 13:55:22.928776 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
I20250627 13:55:22.945739 9620 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 18c6bddc499b4363a95de5d2578e4a07)
I20250627 13:55:22.945976 9620 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:22.946664 9605 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 18c6bddc499b4363a95de5d2578e4a07)
I20250627 13:55:22.946732 9620 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:22.946883 9605 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:22.947580 9605 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:22.947635 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" candidate_term: 2 candidate_status { last_received { term: 1 index: 159 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:22.947933 9302 raft_consensus.cc:2408] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e07fa0ee9b1b4e28b800d1ac6da68e17 for term 2 because replica has last-logged OpId of term: 1 index: 160, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 159.
I20250627 13:55:22.948767 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:22.949013 9173 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 1.
W20250627 13:55:22.949007 9239 leader_election.cc:336] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
W20250627 13:55:22.949262 9109 leader_election.cc:336] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
I20250627 13:55:22.949592 9109 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07, 8064817311e242618056800a0ca537dd
I20250627 13:55:22.949864 9620 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:22.950160 9240 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters: 18c6bddc499b4363a95de5d2578e4a07
I20250627 13:55:22.950531 9564 raft_consensus.cc:2802] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250627 13:55:22.950645 9564 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 18c6bddc499b4363a95de5d2578e4a07)
I20250627 13:55:22.950788 9564 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:55:22.953565 9263 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35262: Illegal state: replica 8064817311e242618056800a0ca537dd is not leader of this config: current role FOLLOWER
I20250627 13:55:22.953778 9564 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:22.955116 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "8064817311e242618056800a0ca537dd" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17"
I20250627 13:55:22.955217 9564 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 election: Requested vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:22.955371 9173 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:55:22.957873 9173 raft_consensus.cc:2466] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8064817311e242618056800a0ca537dd in term 2.
I20250627 13:55:22.958359 9240 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17; no voters:
I20250627 13:55:22.958812 9564 raft_consensus.cc:2802] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:55:22.959080 9564 raft_consensus.cc:695] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 2 LEADER]: Becoming Leader. State: Replica: 8064817311e242618056800a0ca537dd, State: Running, Role: LEADER
W20250627 13:55:22.959473 9239 leader_election.cc:336] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
I20250627 13:55:22.959443 9564 consensus_queue.cc:237] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.160, Last appended by leader: 160, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:22.961838 8894 catalog_manager.cc:5582] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd reported cstate change: term changed from 1 to 2, leader changed from 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193) to 8064817311e242618056800a0ca537dd (127.8.167.195). New cstate: current_term: 2 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: UNKNOWN } } }
W20250627 13:55:22.977463 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
I20250627 13:55:22.977730 9173 raft_consensus.cc:1273] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 2 FOLLOWER]: Refusing update from remote peer 8064817311e242618056800a0ca537dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 159. Preceding OpId from leader: term: 2 index: 162. (index mismatch)
I20250627 13:55:22.978312 9605 consensus_queue.cc:1035] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 158, Time since last communication: 0.000s
W20250627 13:55:22.978740 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:22.984706 9441 mvcc.cc:204] Tried to move back new op lower bound from 7172229214110580736 to 7172229214043664384. Current Snapshot: MvccSnapshot[applied={T|T < 7172229214110580736 or (T in {7172229214110580736})}]
I20250627 13:55:22.987534 9438 mvcc.cc:204] Tried to move back new op lower bound from 7172229214110580736 to 7172229214043664384. Current Snapshot: MvccSnapshot[applied={T|T < 7172229214110580736 or (T in {7172229214110580736})}]
W20250627 13:55:22.995548 9123 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34444: Illegal state: replica e07fa0ee9b1b4e28b800d1ac6da68e17 is not leader of this config: current role FOLLOWER
W20250627 13:55:23.125011 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.152195 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.188344 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.222014 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.222278 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.476594 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:23.636014 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:23.663767 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:23.703691 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:23.776659 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:23.776859 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:24.011395 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:24.121045 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:24.210534 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:24.210924 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:24.261667 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:24.336922 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250627 13:55:24.408290 9634 consensus_queue.cc:579] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.014s)
I20250627 13:55:24.421144 9631 consensus_queue.cc:579] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.013s)
W20250627 13:55:24.548977 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:24.626164 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:24.627583 9625 consensus_queue.cc:579] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.001s)
I20250627 13:55:24.650097 9621 consensus_queue.cc:579] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.013s)
W20250627 13:55:24.708148 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:24.737036 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:24.761030 9402 consensus_queue.cc:579] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.122s)
W20250627 13:55:24.790956 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:24.859683 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:25.053560 9625 consensus_queue.cc:579] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.094s)
W20250627 13:55:25.060935 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:25.182129 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.185423 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.246724 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.308880 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.414793 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.576385 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:25.672591 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250627 13:55:25.737545 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250627 13:55:25.783252 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250627 13:55:25.818884 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250627 13:55:25.915977 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 8871
I20250627 13:55:25.917281 8932 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:55:26.024158 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250627 13:55:26.078358 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20250627 13:55:26.159513 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 8871
W20250627 13:55:26.174254 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.211707 9349 heartbeater.cc:646] Failed to heartbeat to 127.8.167.254:41403 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.8.167.254:41403 (error 108)
I20250627 13:55:26.212337 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:41403
--webserver_interface=127.8.167.254
--webserver_port=35337
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.167.254:41403 with env {}
W20250627 13:55:26.293284 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.312139 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.366649 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.582731 9599 heartbeater.cc:646] Failed to heartbeat to 127.8.167.254:41403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.167.254:41403: connect: Connection refused (error 111)
W20250627 13:55:26.593708 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.670958 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250627 13:55:26.671190 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250627 13:55:26.815070 9219 heartbeater.cc:646] Failed to heartbeat to 127.8.167.254:41403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.167.254:41403: connect: Connection refused (error 111)
W20250627 13:55:26.821523 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250627 13:55:26.842056 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250627 13:55:26.897831 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250627 13:55:26.978142 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:26.978724 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:26.978910 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:26.991874 9642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:55:26.992164 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:26.992336 9642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:55:26.992455 9642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:55:27.008776 9642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.167.254:41403
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.167.254
--webserver_port=35337
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:27.011353 9642 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:27.014259 9642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:27.050594 9650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:27.065476 9642 server_base.cc:1048] running on GCE node
W20250627 13:55:27.066756 9652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:27.070979 9649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:27.072237 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250627 13:55:27.095984 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20250627 13:55:27.190339 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250627 13:55:27.211392 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20250627 13:55:27.222144 9642 hybrid_clock.cc:648] HybridClock initialized: now 1751032527221972 us; error 96 us; skew 500 ppm
I20250627 13:55:27.222749 9642 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:27.237866 9642 webserver.cc:469] Webserver started at http://127.8.167.254:35337/ using document root <none> and password file <none>
I20250627 13:55:27.239243 9642 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:27.239449 9642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250627 13:55:27.264729 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250627 13:55:27.337621 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20250627 13:55:27.344475 9642 fs_manager.cc:714] Time spent opening directory manager: real 0.102s user 0.040s sys 0.055s
I20250627 13:55:27.354661 9658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:27.356894 9642 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.005s
I20250627 13:55:27.357383 9642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
uuid: "93ef319ee202403688689c9c1fed179c"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:27.358733 9642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250627 13:55:27.386440 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20250627 13:55:27.416819 9642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:27.418180 9642 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:27.418519 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250627 13:55:27.443795 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250627 13:55:27.674655 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250627 13:55:27.690245 9239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20250627 13:55:27.699954 9109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20250627 13:55:27.761684 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250627 13:55:27.827461 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250627 13:55:27.878062 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20250627 13:55:27.886606 9642 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.254:41403
I20250627 13:55:27.886889 9710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.254:41403 every 8 connection(s)
I20250627 13:55:27.888895 9642 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
I20250627 13:55:27.890893 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 9642
I20250627 13:55:27.924736 9711 sys_catalog.cc:263] Verifying existing consensus state
I20250627 13:55:27.958684 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Bootstrap starting.
W20250627 13:55:27.965696 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250627 13:55:27.973001 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20250627 13:55:28.004433 9711 log.cc:826] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:55:28.037782 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=11 ignored=0} mutations{seen=13 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:55:28.039170 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Bootstrap complete.
I20250627 13:55:28.060729 9711 raft_consensus.cc:357] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:28.061597 9711 raft_consensus.cc:738] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ef319ee202403688689c9c1fed179c, State: Initialized, Role: FOLLOWER
I20250627 13:55:28.062397 9711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:28.062613 9711 raft_consensus.cc:397] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:55:28.062762 9711 raft_consensus.cc:491] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:55:28.062927 9711 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:55:28.067025 9711 raft_consensus.cc:513] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:28.067734 9711 leader_election.cc:304] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ef319ee202403688689c9c1fed179c; no voters:
I20250627 13:55:28.131247 9711 leader_election.cc:290] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 13:55:28.131326 9714 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 2 FOLLOWER]: Leader election won for term 2
I20250627 13:55:28.136168 9714 raft_consensus.cc:695] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [term 2 LEADER]: Becoming Leader. State: Replica: 93ef319ee202403688689c9c1fed179c, State: Running, Role: LEADER
I20250627 13:55:28.136940 9714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 14, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } }
I20250627 13:55:28.140892 9711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:55:28.248966 9349 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:28.303653 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ef319ee202403688689c9c1fed179c. Latest consensus state: current_term: 2 leader_uuid: "93ef319ee202403688689c9c1fed179c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } } }
I20250627 13:55:28.304018 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: This master's current role is: LEADER
I20250627 13:55:28.308725 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93ef319ee202403688689c9c1fed179c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ef319ee202403688689c9c1fed179c" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 41403 } } }
I20250627 13:55:28.309103 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c [sys.catalog]: This master's current role is: LEADER
W20250627 13:55:28.309121 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250627 13:55:28.375658 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250627 13:55:28.390954 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250627 13:55:28.416908 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250627 13:55:28.449894 9109 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.195:43925, user_credentials={real_user=slave}} blocked reactor thread for 58524.4us
I20250627 13:55:28.452913 9729 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
W20250627 13:55:28.473685 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20250627 13:55:28.479396 9729 catalog_manager.cc:671] Loaded metadata for table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4]
I20250627 13:55:28.483148 9729 tablet_loader.cc:96] loaded metadata for tablet 22aa1c36043f4e8781bac641bd1035aa (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
I20250627 13:55:28.484228 9729 tablet_loader.cc:96] loaded metadata for tablet 82aebc7eff8c4c8a86eb994bf24bd99c (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
I20250627 13:55:28.484969 9729 tablet_loader.cc:96] loaded metadata for tablet 9ef29c61340f4869b8703e67f2e32f14 (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
I20250627 13:55:28.485453 9729 tablet_loader.cc:96] loaded metadata for tablet ab269b1c09ed41b0b51072f18c2462d8 (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
W20250627 13:55:28.494715 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20250627 13:55:28.486208 9729 tablet_loader.cc:96] loaded metadata for tablet c370dad30118471cbdc7094f7209aad5 (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
I20250627 13:55:28.499774 9729 tablet_loader.cc:96] loaded metadata for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 (table test-workload [id=7cb5f868c8604d64bbfc94c6474784c4])
I20250627 13:55:28.502219 9729 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:55:28.504519 9729 catalog_manager.cc:1261] Loaded cluster ID: 399dc6bde91d44c9b89f31112cf9df43
I20250627 13:55:28.505967 9729 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:55:28.513757 9729 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:55:28.517877 9729 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 93ef319ee202403688689c9c1fed179c: Loaded TSK: 0
I20250627 13:55:28.520205 9729 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:55:28.720957 9599 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
W20250627 13:55:28.723695 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20250627 13:55:28.730312 9676 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" instance_seqno: 1751032520958512) as {username='slave'} at 127.8.167.196:50001; Asking this server to re-register.
I20250627 13:55:28.740051 9599 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:28.740345 9599 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:28.742019 9676 ts_manager.cc:194] Registered new tserver with Master: 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697)
W20250627 13:55:28.812222 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20250627 13:55:28.853044 9219 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:28.859707 9676 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" instance_seqno: 1751032517503710) as {username='slave'} at 127.8.167.194:55313; Asking this server to re-register.
I20250627 13:55:28.863672 9219 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:28.863969 9219 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:28.873118 9676 ts_manager.cc:194] Registered new tserver with Master: e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
W20250627 13:55:28.947618 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250627 13:55:29.001159 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250627 13:55:29.106318 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250627 13:55:29.113571 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250627 13:55:29.260241 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20250627 13:55:29.291285 9676 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8064817311e242618056800a0ca537dd" instance_seqno: 1751032517956930) as {username='slave'} at 127.8.167.195:60271; Asking this server to re-register.
I20250627 13:55:29.292236 9349 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:29.292534 9349 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:29.294443 9676 ts_manager.cc:194] Registered new tserver with Master: 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
W20250627 13:55:29.325218 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:29.440004 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:29.577248 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20250627 13:55:29.599208 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:29.660606 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:29.735005 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:29.832809 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:29.924870 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:30.102175 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:30.178294 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:30.185604 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:30.277207 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:30.412614 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:30.437196 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:30.640233 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:30.677639 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:30.738498 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:30.844465 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:30.934352 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:30.973582 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20250627 13:55:31.006754 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:45719
--local_ip_for_outbound_sockets=127.8.167.193
--tserver_master_addrs=127.8.167.254:41403
--webserver_port=34195
--webserver_interface=127.8.167.193
--builtin_ntp_servers=127.8.167.212:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=2
--enable_log_gc=false with env {}
W20250627 13:55:31.123709 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:31.215240 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:31.270481 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:31.415475 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:31.502494 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:31.594265 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:31.684139 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:31.815268 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:31.824928 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:31.835433 9743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:31.836927 9743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:31.837103 9743 flags.cc:425] Enabled unsafe flag: --enable_log_gc=false
W20250627 13:55:31.837229 9743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:31.851430 9743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:31.851794 9743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:55:31.870064 9743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=2
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:45719
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=34195
--enable_log_gc=false
--tserver_master_addrs=127.8.167.254:41403
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:31.872537 9743 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:31.875303 9743 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:31.955606 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:31.960404 9748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:32.045459 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250627 13:55:32.048703 9749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:32.066720 9751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:32.085839 9743 server_base.cc:1048] running on GCE node
I20250627 13:55:32.086829 9743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:32.127686 9743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:32.131079 9743 hybrid_clock.cc:648] HybridClock initialized: now 1751032532130724 us; error 303 us; skew 500 ppm
I20250627 13:55:32.131522 9743 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:32.168797 9743 webserver.cc:469] Webserver started at http://127.8.167.193:34195/ using document root <none> and password file <none>
I20250627 13:55:32.170784 9743 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:32.171114 9743 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:32.212673 9743 fs_manager.cc:714] Time spent opening directory manager: real 0.038s user 0.008s sys 0.010s
W20250627 13:55:32.229216 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20250627 13:55:32.248881 9757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:32.281430 9743 fs_manager.cc:730] Time spent opening block manager: real 0.065s user 0.008s sys 0.011s
I20250627 13:55:32.281735 9743 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "18c6bddc499b4363a95de5d2578e4a07"
format_stamp: "Formatted at 2025-06-27 13:55:16 on dist-test-slave-1xs4"
I20250627 13:55:32.283874 9743 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250627 13:55:32.320552 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:32.336054 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250627 13:55:32.389402 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250627 13:55:32.397385 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20250627 13:55:32.399562 9743 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:32.400754 9743 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:32.401150 9743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:32.411630 9743 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:32.436872 9764 ts_tablet_manager.cc:542] Loading tablet metadata (0/6 complete)
I20250627 13:55:32.465968 9743 ts_tablet_manager.cc:579] Loaded tablet metadata (6 total tablets, 6 live tablets)
I20250627 13:55:32.466131 9743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.043s user 0.004s sys 0.008s
I20250627 13:55:32.466259 9743 ts_tablet_manager.cc:594] Registering tablets (0/6 complete)
I20250627 13:55:32.492221 9764 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:32.514180 9743 ts_tablet_manager.cc:610] Registered 6 tablets
I20250627 13:55:32.514362 9743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.048s user 0.012s sys 0.006s
W20250627 13:55:32.617126 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250627 13:55:32.639199 9764 log.cc:826] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Log is configured to *not* fsync() on all Append() calls
W20250627 13:55:32.703037 9109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20250627 13:55:32.713498 9239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20250627 13:55:32.771296 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250627 13:55:32.818866 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20250627 13:55:32.932863 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20250627 13:55:33.017695 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250627 13:55:33.050308 9764 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=1338 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:55:33.051065 9764 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:33.054989 9764 ts_tablet_manager.cc:1397] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.563s user 0.258s sys 0.051s
I20250627 13:55:33.063205 9764 raft_consensus.cc:357] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.064997 9764 raft_consensus.cc:738] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:33.066092 9764 consensus_queue.cc:260] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.076067 9764 ts_tablet_manager.cc:1428] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.021s user 0.009s sys 0.009s
I20250627 13:55:33.076606 9764 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
W20250627 13:55:33.178685 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250627 13:55:33.324796 9789 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:33.325119 9789 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.331557 9789 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
W20250627 13:55:33.342381 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20250627 13:55:33.495982 9764 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=157 ignored=0} inserts{seen=1372 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250627 13:55:33.496091 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20250627 13:55:33.496843 9764 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:33.498615 9764 ts_tablet_manager.cc:1397] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.422s user 0.325s sys 0.035s
I20250627 13:55:33.500029 9764 raft_consensus.cc:357] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.500456 9764 raft_consensus.cc:738] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:33.500809 9764 consensus_queue.cc:260] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 157, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.501727 9764 ts_tablet_manager.cc:1428] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250627 13:55:33.502135 9764 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
W20250627 13:55:33.522706 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20250627 13:55:33.540392 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20250627 13:55:33.562582 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:33.584693 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:33.585724 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:33.586844 9789 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:33.649061 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20250627 13:55:33.863478 9789 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:33.863677 9789 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.864436 9789 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:33.866966 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 157 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:33.867875 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 157 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:33.869102 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:33.869632 9789 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:33.902266 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20250627 13:55:33.905640 9789 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:33.905828 9789 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.906545 9789 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:33.907232 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:33.908555 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:33.909184 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:33.910055 9789 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:33.965499 9764 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=1369 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:55:33.966233 9764 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:33.967638 9764 ts_tablet_manager.cc:1397] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.466s user 0.333s sys 0.021s
I20250627 13:55:33.968607 9764 raft_consensus.cc:357] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.968880 9764 raft_consensus.cc:738] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:33.969194 9764 consensus_queue.cc:260] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:33.970222 9764 ts_tablet_manager.cc:1428] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250627 13:55:33.970577 9764 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
W20250627 13:55:33.991026 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20250627 13:55:34.023627 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20250627 13:55:34.131230 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20250627 13:55:34.197193 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20250627 13:55:34.220178 9882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:45719 every 8 connection(s)
I20250627 13:55:34.221197 9743 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:45719
I20250627 13:55:34.223771 9743 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:55:34.231874 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 9743
I20250627 13:55:34.242002 9789 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:34.242209 9789 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:34.242959 9789 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:34.246498 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:34.279943 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:34.280998 9760 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:34.281445 9789 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:34.294657 9883 heartbeater.cc:344] Connected to a master server at 127.8.167.254:41403
I20250627 13:55:34.295140 9883 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:34.296532 9883 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:34.300508 9676 ts_manager.cc:194] Registered new tserver with Master: 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719)
I20250627 13:55:34.304164 9676 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:39909
I20250627 13:55:34.310609 9789 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:34.311105 9789 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:34.312546 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 157 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:34.313925 9789 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:34.319092 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 157 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:34.324076 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:34.324704 9789 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:34.365324 9789 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:34.365533 9789 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:34.366289 9789 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:34.367074 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:34.369747 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:34.370498 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:34.371001 9789 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:34.512753 9825 raft_consensus.cc:1215] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.158->[1.159-1.760] Dedup: 1.160->[1.161-1.760]
I20250627 13:55:34.514216 9883 heartbeater.cc:499] Master 127.8.167.254:41403 was elected leader, sending a full tablet report...
I20250627 13:55:34.538862 9764 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=160 ignored=0} inserts{seen=1260 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:55:34.548331 9764 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:34.549853 9764 ts_tablet_manager.cc:1397] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 0.579s user 0.249s sys 0.038s
I20250627 13:55:34.551141 9764 raft_consensus.cc:357] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:34.554340 9764 raft_consensus.cc:738] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:34.554742 9764 consensus_queue.cc:260] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:34.559686 9764 ts_tablet_manager.cc:1428] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250627 13:55:34.565750 9764 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:34.601004 9829 raft_consensus.cc:1215] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.156->[1.157-1.759] Dedup: 1.157->[1.158-1.759]
W20250627 13:55:34.815501 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250627 13:55:35.819345 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 111: this message will repeat every 5th retry.
W20250627 13:55:36.230657 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250627 13:55:36.598474 9895 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:36.598726 9895 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:36.599886 9895 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:36.603118 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:36.606498 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:36.611867 9829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 1) took 2251 ms. Trace:
I20250627 13:55:36.622402 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:36.622649 9787 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57736 (request call id 4) took 2116 ms. Trace:
I20250627 13:55:36.656664 9895 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:36.656800 9761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 47988.7us
I20250627 13:55:36.667943 9787 rpcz_store.cc:276] 0627 13:55:34.485886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:34.486593 (+ 707us) service_pool.cc:225] Handling call
0627 13:55:36.602359 (+2115766us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x616000017b54. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:36.602591 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4849290112}
I20250627 13:55:36.831087 9894 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:36.831291 9894 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:36.836433 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:36.837316 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:36.847609 9760 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:36.846134 9894 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:36.873405 9895 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:36.892938 9789 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:36.893136 9789 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:36.894536 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:36.896090 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:36.898311 9789 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:36.902184 9825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 2) took 2499 ms. Trace:
I20250627 13:55:36.903649 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:36.889567 9828 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 2) took 2478 ms. Trace:
I20250627 13:55:36.907431 9789 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.108834 9789 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.109071 9789 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.110327 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.110356 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.112340 9789 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.113060 9760 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.113850 9789 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:37.154136 9240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 54633us
I20250627 13:55:37.270555 9789 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:37.270831 9789 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.274544 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.279790 9789 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.285751 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.288141 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.288859 9789 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.291379 9789 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.291613 9789 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.292349 9789 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.293406 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.297585 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.299861 9760 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.300882 9789 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.510242 9893 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.510484 9893 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.545708 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.545879 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.560570 9893 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.562789 9760 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.563390 9893 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.576026 9826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 3) took 2931 ms. Trace:
I20250627 13:55:37.635470 9893 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.635670 9893 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.637493 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.637754 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.641075 9893 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.662858 9760 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.663180 9893 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.676436 9893 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:37.676644 9893 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.679723 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.680193 9893 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.681939 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.698788 9761 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.701506 9893 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.714304 9893 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.714488 9893 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.715107 9893 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.721376 9304 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.728307 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.741281 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.783378 9893 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.886319 9893 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:37.886538 9893 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:37.921721 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:37.922751 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:37.945397 9760 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:37.955281 9893 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:37.957794 9893 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:38.023710 9893 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:38.023931 9893 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:38.024724 9893 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:38.025712 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
W20250627 13:55:38.102977 9760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.167.195:43925, user_credentials={real_user=slave}} blocked reactor thread for 164140us
I20250627 13:55:38.104153 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:38.105706 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:38.108449 9912 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:38.342180 9912 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:38.345085 9912 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:38.345952 9912 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:38.347052 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:38.347818 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:38.350409 9760 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:38.359917 9912 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:38.392918 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 116: this message will repeat every 5th retry.
I20250627 13:55:38.505184 9912 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:38.536597 9912 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:38.537313 9912 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:38.537707 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:38.543310 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:38.546533 9912 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:38.551921 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:38.573380 9912 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:38.581734 9912 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:38.581920 9912 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:38.585150 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
W20250627 13:55:38.683120 9761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 139127us
I20250627 13:55:38.730147 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:38.773133 9761 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:38.774664 9913 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:38.849647 9913 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:38.853744 9913 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:38.877574 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:38.884491 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 760 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:38.876993 9913 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:38.893584 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:38.894359 9913 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:39.172499 9913 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:39.198521 9913 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:39.199285 9913 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:39.200637 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:39.206001 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 756 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:39.212185 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:39.225989 9913 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:36.924409 9828 rpcz_store.cc:276] 0627 13:55:34.345214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:34.356960 (+ 11746us) service_pool.cc:225] Handling call
0627 13:55:34.491637 (+134677us) raft_consensus.cc:1469] Updating replica for 596 ops
0627 13:55:34.581236 (+ 89599us) raft_consensus.cc:1532] Early marking committed up to index 0
0627 13:55:34.581242 (+ 6us) raft_consensus.cc:1537] Triggering prepare for 596 ops
0627 13:55:34.581442 (+ 200us) write_op.cc:270] Start()
0627 13:55:34.581476 (+ 34us) write_op.cc:276] Timestamp: P: 1751032522654154 usec, L: 0
0627 13:55:34.611120 (+ 29644us) write_op.cc:270] Start()
0627 13:55:34.611171 (+ 51us) write_op.cc:276] Timestamp: P: 1751032522662446 usec, L: 0
0627 13:55:34.611447 (+ 276us) write_op.cc:270] Start()
0627 13:55:34.611488 (+ 41us) write_op.cc:276] Timestamp: P: 1751032522990940 usec, L: 0
0627 13:55:34.611679 (+ 191us) write_op.cc:270] Start()
0627 13:55:34.611700 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523018070 usec, L: 0
0627 13:55:34.611858 (+ 158us) write_op.cc:270] Start()
0627 13:55:34.611879 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523022231 usec, L: 0
0627 13:55:34.612051 (+ 172us) write_op.cc:270] Start()
0627 13:55:34.612073 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523048856 usec, L: 0
0627 13:55:34.612281 (+ 208us) write_op.cc:270] Start()
0627 13:55:34.612326 (+ 45us) write_op.cc:276] Timestamp: P: 1751032523074539 usec, L: 0
0627 13:55:34.612483 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.612504 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523076981 usec, L: 0
0627 13:55:34.612661 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.612679 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523111791 usec, L: 0
0627 13:55:34.612833 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.612852 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523112936 usec, L: 0
0627 13:55:34.612988 (+ 136us) write_op.cc:270] Start()
0627 13:55:34.613005 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523113742 usec, L: 0
0627 13:55:34.613146 (+ 141us) write_op.cc:270] Start()
0627 13:55:34.613177 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523115813 usec, L: 0
0627 13:55:34.613326 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.613346 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523151346 usec, L: 0
0627 13:55:34.613501 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.613522 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523178615 usec, L: 0
0627 13:55:34.613684 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.613706 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523180011 usec, L: 0
0627 13:55:34.630131 (+ 16425us) write_op.cc:270] Start()
0627 13:55:34.630150 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523196995 usec, L: 0
0627 13:55:34.630317 (+ 167us) write_op.cc:270] Start()
0627 13:55:34.630346 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523210151 usec, L: 0
0627 13:55:34.630497 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.630517 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523226844 usec, L: 0
0627 13:55:34.630638 (+ 121us) write_op.cc:270] Start()
0627 13:55:34.630656 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523239937 usec, L: 0
0627 13:55:34.630777 (+ 121us) write_op.cc:270] Start()
0627 13:55:34.630793 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523254329 usec, L: 0
0627 13:55:34.630940 (+ 147us) write_op.cc:270] Start()
0627 13:55:34.630959 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523274995 usec, L: 0
0627 13:55:34.631103 (+ 144us) write_op.cc:270] Start()
0627 13:55:34.631121 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523292411 usec, L: 0
0627 13:55:34.637044 (+ 5923us) write_op.cc:270] Start()
0627 13:55:34.637080 (+ 36us) write_op.cc:276] Timestamp: P: 1751032523308165 usec, L: 0
0627 13:55:34.637282 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.637306 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523310493 usec, L: 0
0627 13:55:34.637502 (+ 196us) write_op.cc:270] Start()
0627 13:55:34.637524 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523314259 usec, L: 0
0627 13:55:34.637713 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.637737 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523328030 usec, L: 0
0627 13:55:34.641605 (+ 3868us) write_op.cc:270] Start()
0627 13:55:34.641631 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523344954 usec, L: 0
0627 13:55:34.641788 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.641807 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523366052 usec, L: 0
0627 13:55:34.641940 (+ 133us) write_op.cc:270] Start()
0627 13:55:34.641959 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523382365 usec, L: 0
0627 13:55:34.642107 (+ 148us) write_op.cc:270] Start()
0627 13:55:34.642125 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523390024 usec, L: 0
0627 13:55:34.642396 (+ 271us) write_op.cc:270] Start()
0627 13:55:34.642417 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523397526 usec, L: 0
0627 13:55:34.642559 (+ 142us) write_op.cc:270] Start()
0627 13:55:34.642588 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523426679 usec, L: 0
0627 13:55:34.642770 (+ 182us) write_op.cc:270] Start()
0627 13:55:34.642791 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523439108 usec, L: 0
0627 13:55:34.649412 (+ 6621us) write_op.cc:270] Start()
0627 13:55:34.649442 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523445850 usec, L: 0
0627 13:55:34.649639 (+ 197us) write_op.cc:270] Start()
0627 13:55:34.649661 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523454803 usec, L: 0
0627 13:55:34.649804 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.649828 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523463934 usec, L: 0
0627 13:55:34.649988 (+ 160us) write_op.cc:270] Start()
0627 13:55:34.650080 (+ 92us) write_op.cc:276] Timestamp: P: 1751032523491180 usec, L: 0
0627 13:55:34.650309 (+ 229us) write_op.cc:270] Start()
0627 13:55:34.650332 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523514950 usec, L: 0
0627 13:55:34.650485 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.650505 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523521102 usec, L: 0
0627 13:55:34.650639 (+ 134us) write_op.cc:270] Start()
0627 13:55:34.650657 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523525901 usec, L: 0
0627 13:55:34.650783 (+ 126us) write_op.cc:270] Start()
0627 13:55:34.650802 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523547443 usec, L: 0
0627 13:55:34.670401 (+ 19599us) write_op.cc:270] Start()
0627 13:55:34.670424 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523573640 usec, L: 0
0627 13:55:34.670591 (+ 167us) write_op.cc:270] Start()
0627 13:55:34.670612 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523574917 usec, L: 0
0627 13:55:34.670751 (+ 139us) write_op.cc:270] Start()
0627 13:55:34.670772 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523603910 usec, L: 0
0627 13:55:34.670934 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.670954 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523614184 usec, L: 0
0627 13:55:34.671123 (+ 169us) write_op.cc:270] Start()
0627 13:55:34.671148 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523620682 usec, L: 0
0627 13:55:34.671330 (+ 182us) write_op.cc:270] Start()
0627 13:55:34.671354 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523629695 usec, L: 0
0627 13:55:34.671504 (+ 150us) write_op.cc:270] Start()
0627 13:55:34.671528 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523661520 usec, L: 0
0627 13:55:34.671677 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.671708 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523666821 usec, L: 0
0627 13:55:34.671897 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.671923 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523683435 usec, L: 0
0627 13:55:34.672243 (+ 320us) write_op.cc:270] Start()
0627 13:55:34.672271 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523716856 usec, L: 0
0627 13:55:34.672433 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.672457 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523720307 usec, L: 0
0627 13:55:34.672622 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.672646 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523726240 usec, L: 0
0627 13:55:34.672833 (+ 187us) write_op.cc:270] Start()
0627 13:55:34.672857 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523740345 usec, L: 0
0627 13:55:34.673044 (+ 187us) write_op.cc:270] Start()
0627 13:55:34.673069 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523772962 usec, L: 0
0627 13:55:34.689537 (+ 16468us) write_op.cc:270] Start()
0627 13:55:34.689583 (+ 46us) write_op.cc:276] Timestamp: P: 1751032523773795 usec, L: 0
0627 13:55:34.690529 (+ 946us) write_op.cc:270] Start()
0627 13:55:34.690571 (+ 42us) write_op.cc:276] Timestamp: P: 1751032523774990 usec, L: 0
0627 13:55:34.690778 (+ 207us) write_op.cc:270] Start()
0627 13:55:34.690797 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523786579 usec, L: 0
0627 13:55:34.690961 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.692785 (+ 1824us) write_op.cc:276] Timestamp: P: 1751032523810950 usec, L: 0
0627 13:55:34.693075 (+ 290us) write_op.cc:270] Start()
0627 13:55:34.693119 (+ 44us) write_op.cc:276] Timestamp: P: 1751032523820170 usec, L: 0
0627 13:55:34.693297 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.693319 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523839222 usec, L: 0
0627 13:55:34.693465 (+ 146us) write_op.cc:270] Start()
0627 13:55:34.693486 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523855696 usec, L: 0
0627 13:55:34.693647 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.693668 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523860907 usec, L: 0
0627 13:55:34.693827 (+ 159us) write_op.cc:270] Start()
0627 13:55:34.693849 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523866833 usec, L: 0
0627 13:55:34.694314 (+ 465us) write_op.cc:270] Start()
0627 13:55:34.694338 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523891158 usec, L: 0
0627 13:55:34.694499 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.694521 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523909197 usec, L: 0
0627 13:55:34.694681 (+ 160us) write_op.cc:270] Start()
0627 13:55:34.694700 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523911909 usec, L: 0
0627 13:55:34.694849 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.694872 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523917543 usec, L: 0
0627 13:55:34.695025 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.695059 (+ 34us) write_op.cc:276] Timestamp: P: 1751032523944198 usec, L: 0
0627 13:55:34.695224 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.695245 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523957947 usec, L: 0
0627 13:55:34.695402 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.695422 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523969808 usec, L: 0
0627 13:55:34.695576 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.695596 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523982824 usec, L: 0
0627 13:55:34.695791 (+ 195us) write_op.cc:270] Start()
0627 13:55:34.695812 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524002968 usec, L: 0
0627 13:55:34.695985 (+ 173us) write_op.cc:270] Start()
0627 13:55:34.696007 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524023028 usec, L: 0
0627 13:55:34.696181 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.696200 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524041396 usec, L: 0
0627 13:55:34.696338 (+ 138us) write_op.cc:270] Start()
0627 13:55:34.696366 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524042941 usec, L: 0
0627 13:55:34.696604 (+ 238us) write_op.cc:270] Start()
0627 13:55:34.696631 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524073591 usec, L: 0
0627 13:55:34.725333 (+ 28702us) write_op.cc:270] Start()
0627 13:55:34.725362 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524085319 usec, L: 0
0627 13:55:34.725551 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.763728 (+ 38177us) write_op.cc:276] Timestamp: P: 1751032524099368 usec, L: 0
0627 13:55:34.774521 (+ 10793us) write_op.cc:270] Start()
0627 13:55:34.774547 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524120895 usec, L: 0
0627 13:55:34.779152 (+ 4605us) write_op.cc:270] Start()
0627 13:55:34.779173 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524137456 usec, L: 0
0627 13:55:34.779325 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.779344 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524143386 usec, L: 0
0627 13:55:34.779515 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.779538 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524152386 usec, L: 0
0627 13:55:34.779676 (+ 138us) write_op.cc:270] Start()
0627 13:55:34.779701 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524153314 usec, L: 0
0627 13:55:34.779859 (+ 158us) write_op.cc:270] Start()
0627 13:55:34.779887 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524181519 usec, L: 0
0627 13:55:34.780057 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.780077 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524191998 usec, L: 0
0627 13:55:34.780255 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.780277 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524201512 usec, L: 0
0627 13:55:34.780456 (+ 179us) write_op.cc:270] Start()
0627 13:55:34.780477 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524226419 usec, L: 0
0627 13:55:34.780678 (+ 201us) write_op.cc:270] Start()
0627 13:55:34.780701 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524231772 usec, L: 0
0627 13:55:34.780862 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.780884 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524244020 usec, L: 0
0627 13:55:34.781086 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.781107 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524265264 usec, L: 0
0627 13:55:34.781695 (+ 588us) write_op.cc:270] Start()
0627 13:55:34.781718 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524277192 usec, L: 0
0627 13:55:34.781877 (+ 159us) write_op.cc:270] Start()
0627 13:55:34.781898 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524285514 usec, L: 0
0627 13:55:34.782072 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.782093 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524298263 usec, L: 0
0627 13:55:34.782254 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.782275 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524311313 usec, L: 0
0627 13:55:34.782461 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.782482 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524319117 usec, L: 0
0627 13:55:34.782638 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.782660 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524345851 usec, L: 0
0627 13:55:34.782806 (+ 146us) write_op.cc:270] Start()
0627 13:55:34.782831 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524351664 usec, L: 0
0627 13:55:34.787669 (+ 4838us) write_op.cc:270] Start()
0627 13:55:34.787697 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524372618 usec, L: 0
0627 13:55:34.787868 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.787887 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524374233 usec, L: 0
0627 13:55:34.790972 (+ 3085us) write_op.cc:270] Start()
0627 13:55:34.792447 (+ 1475us) write_op.cc:276] Timestamp: P: 1751032524399163 usec, L: 0
0627 13:55:34.803738 (+ 11291us) write_op.cc:270] Start()
0627 13:55:34.803775 (+ 37us) write_op.cc:276] Timestamp: P: 1751032524407042 usec, L: 0
0627 13:55:34.811692 (+ 7917us) write_op.cc:270] Start()
0627 13:55:34.811719 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524425921 usec, L: 0
0627 13:55:34.811903 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.811925 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524432886 usec, L: 0
0627 13:55:34.812078 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.812097 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524457323 usec, L: 0
0627 13:55:34.812214 (+ 117us) write_op.cc:270] Start()
0627 13:55:34.812241 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524461612 usec, L: 0
0627 13:55:34.812365 (+ 124us) write_op.cc:270] Start()
0627 13:55:34.812386 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524479661 usec, L: 0
0627 13:55:34.812510 (+ 124us) write_op.cc:270] Start()
0627 13:55:34.812529 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524482859 usec, L: 0
0627 13:55:34.812713 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.812738 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524511660 usec, L: 0
0627 13:55:34.812906 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.812925 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524536401 usec, L: 0
0627 13:55:34.813067 (+ 142us) write_op.cc:270] Start()
0627 13:55:34.813085 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524537470 usec, L: 0
0627 13:55:34.813199 (+ 114us) write_op.cc:270] Start()
0627 13:55:34.813219 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524564014 usec, L: 0
0627 13:55:34.813353 (+ 134us) write_op.cc:270] Start()
0627 13:55:34.813369 (+ 16us) write_op.cc:276] Timestamp: P: 1751032524575275 usec, L: 0
0627 13:55:34.813472 (+ 103us) write_op.cc:270] Start()
0627 13:55:34.813488 (+ 16us) write_op.cc:276] Timestamp: P: 1751032524585016 usec, L: 0
0627 13:55:34.813652 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.813668 (+ 16us) write_op.cc:276] Timestamp: P: 1751032524604229 usec, L: 0
0627 13:55:34.813783 (+ 115us) write_op.cc:270] Start()
0627 13:55:34.813800 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524627039 usec, L: 0
0627 13:55:34.813903 (+ 103us) write_op.cc:270] Start()
0627 13:55:34.813943 (+ 40us) write_op.cc:276] Timestamp: P: 1751032524644871 usec, L: 0
0627 13:55:34.814130 (+ 187us) write_op.cc:270] Start()
0627 13:55:34.814148 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524672506 usec, L: 0
0627 13:55:34.814279 (+ 131us) write_op.cc:270] Start()
0627 13:55:34.814295 (+ 16us) write_op.cc:276] Timestamp: P: 1751032524673340 usec, L: 0
0627 13:55:34.814448 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.814484 (+ 36us) write_op.cc:276] Timestamp: P: 1751032524691610 usec, L: 0
0627 13:55:34.814610 (+ 126us) write_op.cc:270] Start()
0627 13:55:34.814626 (+ 16us) write_op.cc:276] Timestamp: P: 1751032524704084 usec, L: 0
0627 13:55:34.814733 (+ 107us) write_op.cc:270] Start()
0627 13:55:34.814752 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524707869 usec, L: 0
0627 13:55:34.814861 (+ 109us) write_op.cc:270] Start()
0627 13:55:34.814876 (+ 15us) write_op.cc:276] Timestamp: P: 1751032524709318 usec, L: 0
0627 13:55:34.814993 (+ 117us) write_op.cc:270] Start()
0627 13:55:34.815011 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524744193 usec, L: 0
0627 13:55:34.815122 (+ 111us) write_op.cc:270] Start()
0627 13:55:34.815137 (+ 15us) write_op.cc:276] Timestamp: P: 1751032524760972 usec, L: 0
0627 13:55:34.815289 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.815313 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524761800 usec, L: 0
0627 13:55:34.815467 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.815491 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524764691 usec, L: 0
0627 13:55:34.815688 (+ 197us) write_op.cc:270] Start()
0627 13:55:34.815711 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524804843 usec, L: 0
0627 13:55:34.815894 (+ 183us) write_op.cc:270] Start()
0627 13:55:34.815915 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524819297 usec, L: 0
0627 13:55:34.816101 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.816122 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524820107 usec, L: 0
0627 13:55:34.825001 (+ 8879us) write_op.cc:270] Start()
0627 13:55:34.840021 (+ 15020us) write_op.cc:276] Timestamp: P: 1751032524822063 usec, L: 0
0627 13:55:34.862400 (+ 22379us) write_op.cc:270] Start()
0627 13:55:34.862439 (+ 39us) write_op.cc:276] Timestamp: P: 1751032524855144 usec, L: 0
0627 13:55:34.865019 (+ 2580us) write_op.cc:270] Start()
0627 13:55:34.865052 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524867195 usec, L: 0
0627 13:55:34.867305 (+ 2253us) write_op.cc:270] Start()
0627 13:55:34.867334 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524877421 usec, L: 0
0627 13:55:34.868876 (+ 1542us) write_op.cc:270] Start()
0627 13:55:34.868906 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524878094 usec, L: 0
0627 13:55:34.877602 (+ 8696us) write_op.cc:270] Start()
0627 13:55:34.877645 (+ 43us) write_op.cc:276] Timestamp: P: 1751032524906231 usec, L: 0
0627 13:55:34.877908 (+ 263us) write_op.cc:270] Start()
0627 13:55:34.877937 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524914663 usec, L: 0
0627 13:55:34.878138 (+ 201us) write_op.cc:270] Start()
0627 13:55:34.878165 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524932023 usec, L: 0
0627 13:55:34.878362 (+ 197us) write_op.cc:270] Start()
0627 13:55:34.878388 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524941790 usec, L: 0
0627 13:55:34.878560 (+ 172us) write_op.cc:270] Start()
0627 13:55:34.878586 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524951354 usec, L: 0
0627 13:55:34.878811 (+ 225us) write_op.cc:270] Start()
0627 13:55:34.878837 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524975981 usec, L: 0
0627 13:55:34.879045 (+ 208us) write_op.cc:270] Start()
0627 13:55:34.879071 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524989248 usec, L: 0
0627 13:55:34.879237 (+ 166us) write_op.cc:270] Start()
0627 13:55:34.879272 (+ 35us) write_op.cc:276] Timestamp: P: 1751032525001215 usec, L: 0
0627 13:55:34.879467 (+ 195us) write_op.cc:270] Start()
0627 13:55:34.879492 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525027218 usec, L: 0
0627 13:55:34.879693 (+ 201us) write_op.cc:270] Start()
0627 13:55:34.879718 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525035785 usec, L: 0
0627 13:55:34.879891 (+ 173us) write_op.cc:270] Start()
0627 13:55:34.879915 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525039878 usec, L: 0
0627 13:55:34.880060 (+ 145us) write_op.cc:270] Start()
0627 13:55:34.880075 (+ 15us) write_op.cc:276] Timestamp: P: 1751032525064838 usec, L: 0
0627 13:55:34.880222 (+ 147us) write_op.cc:270] Start()
0627 13:55:34.880263 (+ 41us) write_op.cc:276] Timestamp: P: 1751032525085682 usec, L: 0
0627 13:55:34.880458 (+ 195us) write_op.cc:270] Start()
0627 13:55:34.880503 (+ 45us) write_op.cc:276] Timestamp: P: 1751032525101133 usec, L: 0
0627 13:55:34.880729 (+ 226us) write_op.cc:270] Start()
0627 13:55:34.880757 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525102295 usec, L: 0
0627 13:55:34.880930 (+ 173us) write_op.cc:270] Start()
0627 13:55:34.880956 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525130181 usec, L: 0
0627 13:55:34.881121 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.881147 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525156791 usec, L: 0
0627 13:55:34.881358 (+ 211us) write_op.cc:270] Start()
0627 13:55:34.881384 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525158415 usec, L: 0
0627 13:55:34.881558 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.881583 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525161032 usec, L: 0
0627 13:55:34.881763 (+ 180us) write_op.cc:270] Start()
0627 13:55:34.881788 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525177072 usec, L: 0
0627 13:55:34.881958 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.881982 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525207702 usec, L: 0
0627 13:55:34.898217 (+ 16235us) write_op.cc:270] Start()
0627 13:55:34.898257 (+ 40us) write_op.cc:276] Timestamp: P: 1751032525213274 usec, L: 0
0627 13:55:34.898441 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.898459 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525219705 usec, L: 0
0627 13:55:34.898597 (+ 138us) write_op.cc:270] Start()
0627 13:55:34.898619 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525239398 usec, L: 0
0627 13:55:34.898767 (+ 148us) write_op.cc:270] Start()
0627 13:55:34.898784 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525252819 usec, L: 0
0627 13:55:34.898947 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.898978 (+ 31us) write_op.cc:276] Timestamp: P: 1751032525283366 usec, L: 0
0627 13:55:34.899214 (+ 236us) write_op.cc:270] Start()
0627 13:55:34.899242 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525290344 usec, L: 0
0627 13:55:34.899442 (+ 200us) write_op.cc:270] Start()
0627 13:55:34.899467 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525291588 usec, L: 0
0627 13:55:34.899668 (+ 201us) write_op.cc:270] Start()
0627 13:55:34.899695 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525303981 usec, L: 0
0627 13:55:34.899869 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.899923 (+ 54us) write_op.cc:276] Timestamp: P: 1751032525334669 usec, L: 0
0627 13:55:34.900121 (+ 198us) write_op.cc:270] Start()
0627 13:55:34.900148 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525338730 usec, L: 0
0627 13:55:34.906232 (+ 6084us) write_op.cc:270] Start()
0627 13:55:34.906399 (+ 167us) write_op.cc:276] Timestamp: P: 1751032525347529 usec, L: 0
0627 13:55:34.906708 (+ 309us) write_op.cc:270] Start()
0627 13:55:34.906736 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525348161 usec, L: 0
0627 13:55:34.906910 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.906937 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525379822 usec, L: 0
0627 13:55:34.907118 (+ 181us) write_op.cc:270] Start()
0627 13:55:34.907144 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525390632 usec, L: 0
0627 13:55:34.907337 (+ 193us) write_op.cc:270] Start()
0627 13:55:34.907355 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525399164 usec, L: 0
0627 13:55:34.907557 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.907583 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525433258 usec, L: 0
0627 13:55:34.907807 (+ 224us) write_op.cc:270] Start()
0627 13:55:34.907836 (+ 29us) write_op.cc:276] Timestamp: P: 1751032525435004 usec, L: 0
0627 13:55:34.908028 (+ 192us) write_op.cc:270] Start()
0627 13:55:34.908053 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525440262 usec, L: 0
0627 13:55:34.920432 (+ 12379us) write_op.cc:270] Start()
0627 13:55:34.920462 (+ 30us) write_op.cc:276] Timestamp: P: 1751032525446160 usec, L: 0
0627 13:55:34.920660 (+ 198us) write_op.cc:270] Start()
0627 13:55:34.920692 (+ 32us) write_op.cc:276] Timestamp: P: 1751032525476285 usec, L: 0
0627 13:55:34.920854 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.920880 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525494580 usec, L: 0
0627 13:55:34.921032 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.921054 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525506354 usec, L: 0
0627 13:55:34.921282 (+ 228us) write_op.cc:270] Start()
0627 13:55:34.921305 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525508002 usec, L: 0
0627 13:55:34.921473 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.921498 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525522747 usec, L: 0
0627 13:55:34.921716 (+ 218us) write_op.cc:270] Start()
0627 13:55:34.921739 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525583333 usec, L: 0
0627 13:55:34.922488 (+ 749us) write_op.cc:270] Start()
0627 13:55:34.922514 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525587597 usec, L: 0
0627 13:55:34.922693 (+ 179us) write_op.cc:270] Start()
0627 13:55:34.922712 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525589886 usec, L: 0
0627 13:55:34.924276 (+ 1564us) write_op.cc:270] Start()
0627 13:55:34.924305 (+ 29us) write_op.cc:276] Timestamp: P: 1751032525594353 usec, L: 0
0627 13:55:34.925063 (+ 758us) write_op.cc:270] Start()
0627 13:55:34.925086 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525625263 usec, L: 0
0627 13:55:34.925263 (+ 177us) write_op.cc:270] Start()
0627 13:55:34.925291 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525639811 usec, L: 0
0627 13:55:34.925450 (+ 159us) write_op.cc:270] Start()
0627 13:55:34.925471 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525641550 usec, L: 0
0627 13:55:34.925664 (+ 193us) write_op.cc:270] Start()
0627 13:55:34.925684 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525646786 usec, L: 0
0627 13:55:34.929633 (+ 3949us) write_op.cc:270] Start()
0627 13:55:34.929661 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525686769 usec, L: 0
0627 13:55:34.929846 (+ 185us) write_op.cc:270] Start()
0627 13:55:34.929867 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525700811 usec, L: 0
0627 13:55:34.930015 (+ 148us) write_op.cc:270] Start()
0627 13:55:34.930055 (+ 40us) write_op.cc:276] Timestamp: P: 1751032525710729 usec, L: 0
0627 13:55:34.930216 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.930237 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525718550 usec, L: 0
0627 13:55:34.930382 (+ 145us) write_op.cc:270] Start()
0627 13:55:34.930402 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525743962 usec, L: 0
0627 13:55:34.930559 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.930580 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525761144 usec, L: 0
0627 13:55:34.932656 (+ 2076us) write_op.cc:270] Start()
0627 13:55:34.932682 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525768582 usec, L: 0
0627 13:55:34.936297 (+ 3615us) write_op.cc:270] Start()
0627 13:55:34.936323 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525782412 usec, L: 0
0627 13:55:34.936952 (+ 629us) write_op.cc:270] Start()
0627 13:55:34.936980 (+ 28us) write_op.cc:276] Timestamp: P: 17510325
W20250627 13:55:39.302475 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250627 13:55:39.411428 9913 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:39.412700 9913 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:39.414503 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:39.415956 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 772 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:39.429591 9761 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:39.429987 9913 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:39.458801 9913 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:36.903755 9825 rpcz_store.cc:276] 0627 13:55:34.402389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:34.402443 (+ 54us) service_pool.cc:225] Handling call
0627 13:55:34.487478 (+ 85035us) raft_consensus.cc:1469] Updating replica for 602 ops
0627 13:55:34.513954 (+ 26476us) raft_consensus.cc:1532] Early marking committed up to index 0
0627 13:55:34.513964 (+ 10us) raft_consensus.cc:1537] Triggering prepare for 600 ops
0627 13:55:34.514967 (+ 1003us) write_op.cc:270] Start()
0627 13:55:34.515020 (+ 53us) write_op.cc:276] Timestamp: P: 1751032522649810 usec, L: 0
0627 13:55:34.704674 (+189654us) write_op.cc:270] Start()
0627 13:55:34.704696 (+ 22us) write_op.cc:276] Timestamp: P: 1751032522665152 usec, L: 0
0627 13:55:34.778833 (+ 74137us) write_op.cc:270] Start()
0627 13:55:34.778865 (+ 32us) write_op.cc:276] Timestamp: P: 1751032523000123 usec, L: 0
0627 13:55:34.783678 (+ 4813us) write_op.cc:270] Start()
0627 13:55:34.783701 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523040007 usec, L: 0
0627 13:55:34.783876 (+ 175us) write_op.cc:270] Start()
0627 13:55:34.783892 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523042381 usec, L: 0
0627 13:55:34.784083 (+ 191us) write_op.cc:270] Start()
0627 13:55:34.784102 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523047367 usec, L: 0
0627 13:55:34.784317 (+ 215us) write_op.cc:270] Start()
0627 13:55:34.784361 (+ 44us) write_op.cc:276] Timestamp: P: 1751032523075221 usec, L: 0
0627 13:55:34.791018 (+ 6657us) write_op.cc:270] Start()
0627 13:55:34.791038 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523085511 usec, L: 0
0627 13:55:34.791213 (+ 175us) write_op.cc:270] Start()
0627 13:55:34.793276 (+ 2063us) write_op.cc:276] Timestamp: P: 1751032523117906 usec, L: 0
0627 13:55:34.796567 (+ 3291us) write_op.cc:270] Start()
0627 13:55:34.796594 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523121688 usec, L: 0
0627 13:55:34.804070 (+ 7476us) write_op.cc:270] Start()
0627 13:55:34.804097 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523133677 usec, L: 0
0627 13:55:34.804305 (+ 208us) write_op.cc:270] Start()
0627 13:55:34.804327 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523136767 usec, L: 0
0627 13:55:34.806788 (+ 2461us) write_op.cc:270] Start()
0627 13:55:34.806816 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523152625 usec, L: 0
0627 13:55:34.806996 (+ 180us) write_op.cc:270] Start()
0627 13:55:34.807021 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523185484 usec, L: 0
0627 13:55:34.807207 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.807226 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523186233 usec, L: 0
0627 13:55:34.816637 (+ 9411us) write_op.cc:270] Start()
0627 13:55:34.816661 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523189470 usec, L: 0
0627 13:55:34.816823 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.816853 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523209024 usec, L: 0
0627 13:55:34.817040 (+ 187us) write_op.cc:270] Start()
0627 13:55:34.817061 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523227877 usec, L: 0
0627 13:55:34.817200 (+ 139us) write_op.cc:270] Start()
0627 13:55:34.817219 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523238045 usec, L: 0
0627 13:55:34.817397 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.817420 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523253129 usec, L: 0
0627 13:55:34.817591 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.817613 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523266452 usec, L: 0
0627 13:55:34.817759 (+ 146us) write_op.cc:270] Start()
0627 13:55:34.817779 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523291053 usec, L: 0
0627 13:55:34.817934 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.817952 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523301125 usec, L: 0
0627 13:55:34.866529 (+ 48577us) write_op.cc:270] Start()
0627 13:55:34.866557 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523307476 usec, L: 0
0627 13:55:34.868635 (+ 2078us) write_op.cc:270] Start()
0627 13:55:34.868669 (+ 34us) write_op.cc:276] Timestamp: P: 1751032523320343 usec, L: 0
0627 13:55:34.872749 (+ 4080us) write_op.cc:270] Start()
0627 13:55:34.872786 (+ 37us) write_op.cc:276] Timestamp: P: 1751032523328223 usec, L: 0
0627 13:55:34.873004 (+ 218us) write_op.cc:270] Start()
0627 13:55:34.873030 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523348347 usec, L: 0
0627 13:55:34.873255 (+ 225us) write_op.cc:270] Start()
0627 13:55:34.873282 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523378935 usec, L: 0
0627 13:55:34.873457 (+ 175us) write_op.cc:270] Start()
0627 13:55:34.873487 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523388386 usec, L: 0
0627 13:55:34.873652 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.873681 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523390921 usec, L: 0
0627 13:55:34.873851 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.873876 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523396359 usec, L: 0
0627 13:55:34.874098 (+ 222us) write_op.cc:270] Start()
0627 13:55:34.874124 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523419733 usec, L: 0
0627 13:55:34.874331 (+ 207us) write_op.cc:270] Start()
0627 13:55:34.874356 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523441864 usec, L: 0
0627 13:55:34.874520 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.874544 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523445520 usec, L: 0
0627 13:55:34.874722 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.874747 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523452196 usec, L: 0
0627 13:55:34.874931 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.874956 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523461769 usec, L: 0
0627 13:55:34.875155 (+ 199us) write_op.cc:270] Start()
0627 13:55:34.875197 (+ 42us) write_op.cc:276] Timestamp: P: 1751032523507744 usec, L: 0
0627 13:55:34.875402 (+ 205us) write_op.cc:270] Start()
0627 13:55:34.875431 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523509389 usec, L: 0
0627 13:55:34.875605 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.875624 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523524862 usec, L: 0
0627 13:55:34.875780 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.875803 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523539070 usec, L: 0
0627 13:55:34.875959 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.875978 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523541309 usec, L: 0
0627 13:55:34.876155 (+ 177us) write_op.cc:270] Start()
0627 13:55:34.876182 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523573452 usec, L: 0
0627 13:55:34.876366 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.882639 (+ 6273us) write_op.cc:276] Timestamp: P: 1751032523577476 usec, L: 0
0627 13:55:34.893332 (+ 10693us) write_op.cc:270] Start()
0627 13:55:34.893373 (+ 41us) write_op.cc:276] Timestamp: P: 1751032523606301 usec, L: 0
0627 13:55:34.893589 (+ 216us) write_op.cc:270] Start()
0627 13:55:34.893617 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523609889 usec, L: 0
0627 13:55:34.893821 (+ 204us) write_op.cc:270] Start()
0627 13:55:34.893847 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523621015 usec, L: 0
0627 13:55:34.894079 (+ 232us) write_op.cc:270] Start()
0627 13:55:34.894107 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523633795 usec, L: 0
0627 13:55:34.894299 (+ 192us) write_op.cc:270] Start()
0627 13:55:34.894328 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523665965 usec, L: 0
0627 13:55:34.894492 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.894513 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523670179 usec, L: 0
0627 13:55:34.894748 (+ 235us) write_op.cc:270] Start()
0627 13:55:34.894774 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523685339 usec, L: 0
0627 13:55:34.895010 (+ 236us) write_op.cc:270] Start()
0627 13:55:34.895036 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523709485 usec, L: 0
0627 13:55:34.895207 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.895233 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523717254 usec, L: 0
0627 13:55:34.895421 (+ 188us) write_op.cc:270] Start()
0627 13:55:34.895446 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523721187 usec, L: 0
0627 13:55:34.895645 (+ 199us) write_op.cc:270] Start()
0627 13:55:34.895673 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523742154 usec, L: 0
0627 13:55:34.895848 (+ 175us) write_op.cc:270] Start()
0627 13:55:34.895874 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523760282 usec, L: 0
0627 13:55:34.896046 (+ 172us) write_op.cc:270] Start()
0627 13:55:34.896072 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523770576 usec, L: 0
0627 13:55:34.896270 (+ 198us) write_op.cc:270] Start()
0627 13:55:34.896295 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523781215 usec, L: 0
0627 13:55:34.896485 (+ 190us) write_op.cc:270] Start()
0627 13:55:34.896507 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523796177 usec, L: 0
0627 13:55:34.896693 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.896720 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523809472 usec, L: 0
0627 13:55:34.896915 (+ 195us) write_op.cc:270] Start()
0627 13:55:34.896987 (+ 72us) write_op.cc:276] Timestamp: P: 1751032523821932 usec, L: 0
0627 13:55:34.897175 (+ 188us) write_op.cc:270] Start()
0627 13:55:34.897201 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523849529 usec, L: 0
0627 13:55:34.897397 (+ 196us) write_op.cc:270] Start()
0627 13:55:34.897422 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523855303 usec, L: 0
0627 13:55:34.897582 (+ 160us) write_op.cc:270] Start()
0627 13:55:34.897608 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523861463 usec, L: 0
0627 13:55:34.897811 (+ 203us) write_op.cc:270] Start()
0627 13:55:34.897836 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523866057 usec, L: 0
0627 13:55:34.914080 (+ 16244us) write_op.cc:270] Start()
0627 13:55:34.914106 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523893710 usec, L: 0
0627 13:55:34.914300 (+ 194us) write_op.cc:270] Start()
0627 13:55:34.914321 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523913434 usec, L: 0
0627 13:55:34.914472 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.914491 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523916261 usec, L: 0
0627 13:55:34.914632 (+ 141us) write_op.cc:270] Start()
0627 13:55:34.914655 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523919318 usec, L: 0
0627 13:55:34.914816 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.914842 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523946826 usec, L: 0
0627 13:55:34.915003 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.915026 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523963221 usec, L: 0
0627 13:55:34.915190 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.915210 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523984567 usec, L: 0
0627 13:55:34.915363 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.915382 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524000883 usec, L: 0
0627 13:55:34.915568 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.915594 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524029547 usec, L: 0
0627 13:55:34.915765 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.915785 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524031467 usec, L: 0
0627 13:55:34.915917 (+ 132us) write_op.cc:270] Start()
0627 13:55:34.915937 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524043614 usec, L: 0
0627 13:55:34.916093 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.916114 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524051794 usec, L: 0
0627 13:55:34.916322 (+ 208us) write_op.cc:270] Start()
0627 13:55:34.916348 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524084012 usec, L: 0
0627 13:55:34.916493 (+ 145us) write_op.cc:270] Start()
0627 13:55:34.916517 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524088777 usec, L: 0
0627 13:55:34.916680 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.916722 (+ 42us) write_op.cc:276] Timestamp: P: 1751032524089544 usec, L: 0
0627 13:55:34.916907 (+ 185us) write_op.cc:270] Start()
0627 13:55:34.916927 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524119299 usec, L: 0
0627 13:55:34.917098 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.917123 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524140217 usec, L: 0
0627 13:55:34.917285 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.917305 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524147786 usec, L: 0
0627 13:55:34.917444 (+ 139us) write_op.cc:270] Start()
0627 13:55:34.917463 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524162041 usec, L: 0
0627 13:55:34.917612 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.917633 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524163895 usec, L: 0
0627 13:55:34.917787 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.917813 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524182672 usec, L: 0
0627 13:55:34.922292 (+ 4479us) write_op.cc:270] Start()
0627 13:55:34.922323 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524188682 usec, L: 0
0627 13:55:34.922476 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.922497 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524203509 usec, L: 0
0627 13:55:34.922665 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.922685 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524225057 usec, L: 0
0627 13:55:34.930559 (+ 7874us) write_op.cc:270] Start()
0627 13:55:34.930580 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524229834 usec, L: 0
0627 13:55:34.932845 (+ 2265us) write_op.cc:270] Start()
0627 13:55:34.932901 (+ 56us) write_op.cc:276] Timestamp: P: 1751032524239806 usec, L: 0
0627 13:55:34.936952 (+ 4051us) write_op.cc:270] Start()
0627 13:55:34.936980 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524273261 usec, L: 0
0627 13:55:34.938224 (+ 1244us) write_op.cc:270] Start()
0627 13:55:34.938246 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524284846 usec, L: 0
0627 13:55:34.938419 (+ 173us) write_op.cc:270] Start()
0627 13:55:34.938440 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524286369 usec, L: 0
0627 13:55:34.938609 (+ 169us) write_op.cc:270] Start()
0627 13:55:34.938630 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524300490 usec, L: 0
0627 13:55:34.938794 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.938814 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524319172 usec, L: 0
0627 13:55:34.939563 (+ 749us) write_op.cc:270] Start()
0627 13:55:34.939588 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524322380 usec, L: 0
0627 13:55:34.939746 (+ 158us) write_op.cc:270] Start()
0627 13:55:34.939761 (+ 15us) write_op.cc:276] Timestamp: P: 1751032524359524 usec, L: 0
0627 13:55:34.947552 (+ 7791us) write_op.cc:270] Start()
0627 13:55:34.947579 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524360347 usec, L: 0
0627 13:55:34.950797 (+ 3218us) write_op.cc:270] Start()
0627 13:55:34.950820 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524375715 usec, L: 0
0627 13:55:34.951570 (+ 750us) write_op.cc:270] Start()
0627 13:55:34.951597 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524377009 usec, L: 0
0627 13:55:34.951767 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.951790 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524393151 usec, L: 0
0627 13:55:34.965631 (+ 13841us) write_op.cc:270] Start()
0627 13:55:34.965676 (+ 45us) write_op.cc:276] Timestamp: P: 1751032524405138 usec, L: 0
0627 13:55:34.981453 (+ 15777us) write_op.cc:270] Start()
0627 13:55:34.981477 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524427781 usec, L: 0
0627 13:55:34.981659 (+ 182us) write_op.cc:270] Start()
0627 13:55:34.981682 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524437592 usec, L: 0
0627 13:55:34.981859 (+ 177us) write_op.cc:270] Start()
0627 13:55:34.981881 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524461706 usec, L: 0
0627 13:55:34.982070 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.982091 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524463148 usec, L: 0
0627 13:55:34.982246 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.982265 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524482098 usec, L: 0
0627 13:55:34.982403 (+ 138us) write_op.cc:270] Start()
0627 13:55:34.982423 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524483127 usec, L: 0
0627 13:55:34.982578 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.982603 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524514519 usec, L: 0
0627 13:55:34.982765 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.982789 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524537500 usec, L: 0
0627 13:55:34.982926 (+ 137us) write_op.cc:270] Start()
0627 13:55:34.982945 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524556375 usec, L: 0
0627 13:55:34.983094 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.983114 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524579253 usec, L: 0
0627 13:55:34.983271 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.983294 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524581650 usec, L: 0
0627 13:55:34.983465 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.983483 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524582374 usec, L: 0
0627 13:55:34.983648 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.983668 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524612372 usec, L: 0
0627 13:55:34.983794 (+ 126us) write_op.cc:270] Start()
0627 13:55:34.983813 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524618207 usec, L: 0
0627 13:55:35.003485 (+ 19672us) write_op.cc:270] Start()
0627 13:55:35.003543 (+ 58us) write_op.cc:276] Timestamp: P: 1751032524640832 usec, L: 0
0627 13:55:35.004869 (+ 1326us) write_op.cc:270] Start()
0627 13:55:35.004907 (+ 38us) write_op.cc:276] Timestamp: P: 1751032524646568 usec, L: 0
0627 13:55:35.005132 (+ 225us) write_op.cc:270] Start()
0627 13:55:35.005158 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524667517 usec, L: 0
0627 13:55:35.006250 (+ 1092us) write_op.cc:270] Start()
0627 13:55:35.006280 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524713887 usec, L: 0
0627 13:55:35.008079 (+ 1799us) write_op.cc:270] Start()
0627 13:55:35.008110 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524716330 usec, L: 0
0627 13:55:35.022585 (+ 14475us) write_op.cc:270] Start()
0627 13:55:35.022617 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524718510 usec, L: 0
0627 13:55:35.023193 (+ 576us) write_op.cc:270] Start()
0627 13:55:35.023217 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524723669 usec, L: 0
0627 13:55:35.023364 (+ 147us) write_op.cc:270] Start()
0627 13:55:35.023382 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524747777 usec, L: 0
0627 13:55:35.023645 (+ 263us) write_op.cc:270] Start()
0627 13:55:35.023665 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524773195 usec, L: 0
0627 13:55:35.028346 (+ 4681us) write_op.cc:270] Start()
0627 13:55:35.028378 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524777144 usec, L: 0
0627 13:55:35.034890 (+ 6512us) write_op.cc:270] Start()
0627 13:55:35.034919 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524777970 usec, L: 0
0627 13:55:35.043440 (+ 8521us) write_op.cc:270] Start()
0627 13:55:35.043457 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524801884 usec, L: 0
0627 13:55:35.043633 (+ 176us) write_op.cc:270] Start()
0627 13:55:35.043650 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524809224 usec, L: 0
0627 13:55:35.048543 (+ 4893us) write_op.cc:270] Start()
0627 13:55:35.048570 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524822274 usec, L: 0
0627 13:55:35.054169 (+ 5599us) write_op.cc:270] Start()
0627 13:55:35.054226 (+ 57us) write_op.cc:276] Timestamp: P: 1751032524824256 usec, L: 0
0627 13:55:35.055005 (+ 779us) write_op.cc:270] Start()
0627 13:55:35.055034 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524863235 usec, L: 0
0627 13:55:35.055201 (+ 167us) write_op.cc:270] Start()
0627 13:55:35.055222 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524875176 usec, L: 0
0627 13:55:35.055386 (+ 164us) write_op.cc:270] Start()
0627 13:55:35.055410 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524880904 usec, L: 0
0627 13:55:35.056305 (+ 895us) write_op.cc:270] Start()
0627 13:55:35.056329 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524881587 usec, L: 0
0627 13:55:35.063211 (+ 6882us) write_op.cc:270] Start()
0627 13:55:35.063245 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524928334 usec, L: 0
0627 13:55:35.063480 (+ 235us) write_op.cc:270] Start()
0627 13:55:35.063504 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524932122 usec, L: 0
0627 13:55:35.063676 (+ 172us) write_op.cc:270] Start()
0627 13:55:35.063698 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524933100 usec, L: 0
0627 13:55:35.063863 (+ 165us) write_op.cc:270] Start()
0627 13:55:35.063885 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524937379 usec, L: 0
0627 13:55:35.064039 (+ 154us) write_op.cc:270] Start()
0627 13:55:35.064061 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524965948 usec, L: 0
0627 13:55:35.064223 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.064246 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524983010 usec, L: 0
0627 13:55:35.064400 (+ 154us) write_op.cc:270] Start()
0627 13:55:35.064421 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524991623 usec, L: 0
0627 13:55:35.064577 (+ 156us) write_op.cc:270] Start()
0627 13:55:35.064600 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525009619 usec, L: 0
0627 13:55:35.064788 (+ 188us) write_op.cc:270] Start()
0627 13:55:35.064810 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525038773 usec, L: 0
0627 13:55:35.064953 (+ 143us) write_op.cc:270] Start()
0627 13:55:35.064974 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525042045 usec, L: 0
0627 13:55:35.066415 (+ 1441us) write_op.cc:270] Start()
0627 13:55:35.066432 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525044781 usec, L: 0
0627 13:55:35.066556 (+ 124us) write_op.cc:270] Start()
0627 13:55:35.066582 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525057384 usec, L: 0
0627 13:55:35.067197 (+ 615us) write_op.cc:270] Start()
0627 13:55:35.067216 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525094041 usec, L: 0
0627 13:55:35.067378 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.069147 (+ 1769us) write_op.cc:276] Timestamp: P: 1751032525104982 usec, L: 0
0627 13:55:35.073952 (+ 4805us) write_op.cc:270] Start()
0627 13:55:35.073971 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525105795 usec, L: 0
0627 13:55:35.078842 (+ 4871us) write_op.cc:270] Start()
0627 13:55:35.078864 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525127755 usec, L: 0
0627 13:55:35.079017 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.079035 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525152311 usec, L: 0
0627 13:55:35.095823 (+ 16788us) write_op.cc:270] Start()
0627 13:55:35.095849 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525165227 usec, L: 0
0627 13:55:35.096011 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.096032 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525165955 usec, L: 0
0627 13:55:35.096191 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.096211 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525181864 usec, L: 0
0627 13:55:35.096379 (+ 168us) write_op.cc:270] Start()
0627 13:55:35.096400 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525206573 usec, L: 0
0627 13:55:35.096550 (+ 150us) write_op.cc:270] Start()
0627 13:55:35.096571 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525215431 usec, L: 0
0627 13:55:35.096712 (+ 141us) write_op.cc:270] Start()
0627 13:55:35.096731 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525219254 usec, L: 0
0627 13:55:35.096889 (+ 158us) write_op.cc:270] Start()
0627 13:55:35.096912 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525241888 usec, L: 0
0627 13:55:35.097100 (+ 188us) write_op.cc:270] Start()
0627 13:55:35.097119 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525251942 usec, L: 0
0627 13:55:35.097251 (+ 132us) write_op.cc:270] Start()
0627 13:55:35.097268 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525280848 usec, L: 0
0627 13:55:35.097426 (+ 158us) write_op.cc:270] Start()
0627 13:55:35.097445 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525287064 usec, L: 0
0627 13:55:35.097582 (+ 137us) write_op.cc:270] Start()
0627 13:55:35.097601 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525294586 usec, L: 0
0627 13:55:35.097746 (+ 145us) write_op.cc:270] Start()
0627 13:55:35.097768 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525306742 usec, L: 0
0627 13:55:35.097988 (+ 220us) write_op.cc:270] Start()
0627 13:55:35.098015 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525350249 usec, L: 0
0627 13:55:35.106716 (+ 8701us) write_op.cc:270] Start()
0627 13:55:35.106741 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525351295 usec, L: 0
0627 13:55:35.116040 (+ 9299us) write_op.cc:270] Start()
0627 13:55:35.116538 (+ 498us) write_op.cc:276] Timestamp: P: 1751032525353209 usec, L: 0
0627 13:55:35.116777 (+ 239us) write_op.cc:270] Start()
0627 13:55:35.116800 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525354164 usec, L: 0
0627 13:55:35.116953 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.116970 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525386385 usec, L: 0
0627 13:55:35.117081 (+ 111us) write_op.cc:270] Start()
0627 13:55:35.117098 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525393014 usec, L: 0
0627 13:55:35.117236 (+ 138us) write_op.cc:270] Start()
0627 13:55:35.117256 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525406474 usec, L: 0
0627 13:55:35.117410 (+ 154us) write_op.cc:270] Start()
0627 13:55:35.117430 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525434024 usec, L: 0
0627 13:55:35.117582 (+ 152us) write_op.cc:270] Start()
0627 13:55:35.117601 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525435648 usec, L: 0
0627 13:55:35.117773 (+ 172us) write_op.cc:270] Start()
0627 13:55:35.117796 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525447698 usec, L: 0
0627 13:55:35.117959 (+ 163us) write_op.cc:270] Start()
0627 13:55:35.117981 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525448424 usec, L: 0
0627 13:55:35.118174 (+ 193us) write_op.cc:270] Start()
0627 13:55:35.118196 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525480040 usec, L: 0
0627 13:55:35.118358 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.118379 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525496122 usec, L: 0
0627 13:55:35.119070 (+ 691us) write_op.cc:270] Start()
0627 13:55:35.119107 (+ 37us) write_op.cc:276] Timestamp: P: 1751032525502313 usec, L: 0
0627 13:55:35.120230 (+ 1123us) write_op.cc:270] Start()
0627 13:55:35.120251 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525503158 usec, L: 0
0627 13:55:35.120440 (+ 189us) write_op.cc:270] Start()
0627 13:55:35.120461 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525519779 usec, L: 0
0627 13:55:35.120655 (+ 194us) write_op.cc:270] Start()
0627 13:55:35.120683 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525567733 usec, L: 0
0627 13:55:35.120879 (+ 196us) write_op.cc:270] Start()
0627 13:55:35.120905 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525571449 usec, L: 0
0627 13:55:35.121081 (+ 176us) write_op.cc:270] Start()
0627 13:55:35.121104 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525584418 usec, L: 0
0627 13:55:35.130176 (+ 9072us) write_op.cc:270] Start()
0627 13:55:35.130211 (+ 35us) write_op.cc:276] Timestamp: P: 1751032525593450 usec, L: 0
0627 13:55:35.137692 (+ 7481us) write_op.cc:270] Start()
0627 13:55:35.137709 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525622592 usec, L: 0
0627 13:55:35.137834 (+ 125us) write_op.cc:270] Start()
0627 13:55:35.137851 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525635122 usec, L: 0
0627 13:55:35.138023 (+ 172us) write_op.cc:270] Start()
0627 13:55:35.138074 (+ 51us) write_op.cc:276] Timestamp: P: 1751032525649575 usec, L: 0
0627 13:55:35.138201 (+ 127us) write_op.cc:270] Start()
0627 13:55:35.138218 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525650659 usec, L: 0
0627 13:55:35.138356 (+ 138us) write_op.cc:270] Start()
0627 13:55:35.138372 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525689160 usec, L: 0
0627 13:55:35.138624 (+ 252us) write_op.cc:270] Start()
0627 13:55:35.138642 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525706456 usec, L: 0
0627 13:55:35.138791 (+ 149us) write_op.cc:270] Start()
0627 13:55:35.138809 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525712801 usec, L: 0
0627 13:55:35.138931 (+ 122us) write_op.cc:270] Start()
0627 13:55:35.138948 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525718221 usec, L: 0
0627 13:55:35.139063 (+ 115us) write_op.cc:270] Start()
0627 13:55:35.139079 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525740107 usec, L: 0
0627 13:55:35.139186 (+ 107us) write_op.cc:270] Start()
0627 13:55:35.139201 (+ 15us) write_op.cc:276] Timestamp: P: 1751032525762562 usec, L: 0
0627 13:55:35.139334 (+ 133us) write_op.cc:270] Start()
0627 13:55:35.139354 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525771535 usec, L: 0
0627 13:55:35.139478 (+ 124us) write_op.cc:270] Start()
0627 13:55:35.139494 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525802465 usec, L: 0
0627 13:55:35.139609 (+ 115us) write_op.cc:270] Start()
0627 13:55:35.139625 (+ 16us) write_op.cc:276] Timestamp: P: 17510325
I20250627 13:55:39.916227 9913 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:39.918089 9913 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:39.918848 9913 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:39.919381 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:39.919952 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 759 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:39.928982 9760 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:39.940222 9913 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:37.581195 9826 rpcz_store.cc:276] 0627 13:55:34.569498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:34.571167 (+ 1669us) service_pool.cc:225] Handling call
0627 13:55:34.652620 (+ 81453us) raft_consensus.cc:1469] Updating replica for 612 ops
0627 13:55:34.898587 (+245967us) raft_consensus.cc:1532] Early marking committed up to index 0
0627 13:55:34.898592 (+ 5us) raft_consensus.cc:1537] Triggering prepare for 612 ops
0627 13:55:34.898717 (+ 125us) write_op.cc:270] Start()
0627 13:55:34.898736 (+ 19us) write_op.cc:276] Timestamp: P: 1751032522648608 usec, L: 0
0627 13:55:35.034681 (+135945us) write_op.cc:270] Start()
0627 13:55:35.034703 (+ 22us) write_op.cc:276] Timestamp: P: 1751032522663860 usec, L: 0
0627 13:55:35.035557 (+ 854us) write_op.cc:270] Start()
0627 13:55:35.035586 (+ 29us) write_op.cc:276] Timestamp: P: 1751032522990957 usec, L: 0
0627 13:55:35.036240 (+ 654us) write_op.cc:270] Start()
0627 13:55:35.036262 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523016508 usec, L: 0
0627 13:55:35.040114 (+ 3852us) write_op.cc:270] Start()
0627 13:55:35.040148 (+ 34us) write_op.cc:276] Timestamp: P: 1751032523022240 usec, L: 0
0627 13:55:35.040314 (+ 166us) write_op.cc:270] Start()
0627 13:55:35.040335 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523047054 usec, L: 0
0627 13:55:35.040528 (+ 193us) write_op.cc:270] Start()
0627 13:55:35.040576 (+ 48us) write_op.cc:276] Timestamp: P: 1751032523078050 usec, L: 0
0627 13:55:35.040747 (+ 171us) write_op.cc:270] Start()
0627 13:55:35.040771 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523079012 usec, L: 0
0627 13:55:35.041592 (+ 821us) write_op.cc:270] Start()
0627 13:55:35.041617 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523107944 usec, L: 0
0627 13:55:35.041780 (+ 163us) write_op.cc:270] Start()
0627 13:55:35.041801 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523110997 usec, L: 0
0627 13:55:35.041956 (+ 155us) write_op.cc:270] Start()
0627 13:55:35.041977 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523121635 usec, L: 0
0627 13:55:35.042565 (+ 588us) write_op.cc:270] Start()
0627 13:55:35.042595 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523122571 usec, L: 0
0627 13:55:35.042766 (+ 171us) write_op.cc:270] Start()
0627 13:55:35.042789 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523154844 usec, L: 0
0627 13:55:35.042955 (+ 166us) write_op.cc:270] Start()
0627 13:55:35.042976 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523178727 usec, L: 0
0627 13:55:35.043153 (+ 177us) write_op.cc:270] Start()
0627 13:55:35.043175 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523185168 usec, L: 0
0627 13:55:35.043340 (+ 165us) write_op.cc:270] Start()
0627 13:55:35.043362 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523188450 usec, L: 0
0627 13:55:35.043546 (+ 184us) write_op.cc:270] Start()
0627 13:55:35.043576 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523206173 usec, L: 0
0627 13:55:35.043735 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.043754 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523237028 usec, L: 0
0627 13:55:35.044896 (+ 1142us) write_op.cc:270] Start()
0627 13:55:35.044922 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523238129 usec, L: 0
0627 13:55:35.047484 (+ 2562us) write_op.cc:270] Start()
0627 13:55:35.047508 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523250989 usec, L: 0
0627 13:55:35.065478 (+ 17970us) write_op.cc:270] Start()
0627 13:55:35.065504 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523275197 usec, L: 0
0627 13:55:35.065682 (+ 178us) write_op.cc:270] Start()
0627 13:55:35.065701 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523296574 usec, L: 0
0627 13:55:35.074522 (+ 8821us) write_op.cc:270] Start()
0627 13:55:35.074544 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523313303 usec, L: 0
0627 13:55:35.076547 (+ 2003us) write_op.cc:270] Start()
0627 13:55:35.076571 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523318553 usec, L: 0
0627 13:55:35.079306 (+ 2735us) write_op.cc:270] Start()
0627 13:55:35.079328 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523324760 usec, L: 0
0627 13:55:35.079527 (+ 199us) write_op.cc:270] Start()
0627 13:55:35.079545 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523325498 usec, L: 0
0627 13:55:35.079661 (+ 116us) write_op.cc:270] Start()
0627 13:55:35.079678 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523340790 usec, L: 0
0627 13:55:35.079781 (+ 103us) write_op.cc:270] Start()
0627 13:55:35.079795 (+ 14us) write_op.cc:276] Timestamp: P: 1751032523364809 usec, L: 0
0627 13:55:35.079903 (+ 108us) write_op.cc:270] Start()
0627 13:55:35.079918 (+ 15us) write_op.cc:276] Timestamp: P: 1751032523382174 usec, L: 0
0627 13:55:35.080032 (+ 114us) write_op.cc:270] Start()
0627 13:55:35.080048 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523389104 usec, L: 0
0627 13:55:35.080169 (+ 121us) write_op.cc:270] Start()
0627 13:55:35.080195 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523394549 usec, L: 0
0627 13:55:35.080324 (+ 129us) write_op.cc:270] Start()
0627 13:55:35.080341 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523423632 usec, L: 0
0627 13:55:35.080494 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.080513 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523437801 usec, L: 0
0627 13:55:35.080638 (+ 125us) write_op.cc:270] Start()
0627 13:55:35.080655 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523446689 usec, L: 0
0627 13:55:35.080800 (+ 145us) write_op.cc:270] Start()
0627 13:55:35.080822 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523456613 usec, L: 0
0627 13:55:35.080996 (+ 174us) write_op.cc:270] Start()
0627 13:55:35.081021 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523463269 usec, L: 0
0627 13:55:35.081179 (+ 158us) write_op.cc:270] Start()
0627 13:55:35.081247 (+ 68us) write_op.cc:276] Timestamp: P: 1751032523496800 usec, L: 0
0627 13:55:35.081423 (+ 176us) write_op.cc:270] Start()
0627 13:55:35.081442 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523514017 usec, L: 0
0627 13:55:35.081579 (+ 137us) write_op.cc:270] Start()
0627 13:55:35.081598 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523517172 usec, L: 0
0627 13:55:35.081763 (+ 165us) write_op.cc:270] Start()
0627 13:55:35.081782 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523545965 usec, L: 0
0627 13:55:35.081944 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.081963 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523548355 usec, L: 0
0627 13:55:35.082122 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.082140 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523574631 usec, L: 0
0627 13:55:35.082272 (+ 132us) write_op.cc:270] Start()
0627 13:55:35.082299 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523575531 usec, L: 0
0627 13:55:35.082420 (+ 121us) write_op.cc:270] Start()
0627 13:55:35.082438 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523595953 usec, L: 0
0627 13:55:35.082647 (+ 209us) write_op.cc:270] Start()
0627 13:55:35.082668 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523604056 usec, L: 0
0627 13:55:35.082813 (+ 145us) write_op.cc:270] Start()
0627 13:55:35.082833 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523622971 usec, L: 0
0627 13:55:35.082986 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.083018 (+ 32us) write_op.cc:276] Timestamp: P: 1751032523626792 usec, L: 0
0627 13:55:35.083172 (+ 154us) write_op.cc:270] Start()
0627 13:55:35.083190 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523661915 usec, L: 0
0627 13:55:35.083319 (+ 129us) write_op.cc:270] Start()
0627 13:55:35.083336 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523666149 usec, L: 0
0627 13:55:35.083453 (+ 117us) write_op.cc:270] Start()
0627 13:55:35.083469 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523681935 usec, L: 0
0627 13:55:35.083612 (+ 143us) write_op.cc:270] Start()
0627 13:55:35.083629 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523707545 usec, L: 0
0627 13:55:35.083764 (+ 135us) write_op.cc:270] Start()
0627 13:55:35.083782 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523717240 usec, L: 0
0627 13:55:35.083930 (+ 148us) write_op.cc:270] Start()
0627 13:55:35.083953 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523726249 usec, L: 0
0627 13:55:35.084083 (+ 130us) write_op.cc:270] Start()
0627 13:55:35.084100 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523735596 usec, L: 0
0627 13:55:35.084233 (+ 133us) write_op.cc:270] Start()
0627 13:55:35.084249 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523773134 usec, L: 0
0627 13:55:35.084404 (+ 155us) write_op.cc:270] Start()
0627 13:55:35.084421 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523773786 usec, L: 0
0627 13:55:35.084530 (+ 109us) write_op.cc:270] Start()
0627 13:55:35.084546 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523782394 usec, L: 0
0627 13:55:35.084685 (+ 139us) write_op.cc:270] Start()
0627 13:55:35.084702 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523787644 usec, L: 0
0627 13:55:35.101686 (+ 16984us) write_op.cc:270] Start()
0627 13:55:35.101713 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523811373 usec, L: 0
0627 13:55:35.101861 (+ 148us) write_op.cc:270] Start()
0627 13:55:35.122014 (+ 20153us) write_op.cc:276] Timestamp: P: 1751032523824844 usec, L: 0
0627 13:55:35.129564 (+ 7550us) write_op.cc:270] Start()
0627 13:55:35.129587 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523838163 usec, L: 0
0627 13:55:35.135951 (+ 6364us) write_op.cc:270] Start()
0627 13:55:35.135974 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523857452 usec, L: 0
0627 13:55:35.137370 (+ 1396us) write_op.cc:270] Start()
0627 13:55:35.137416 (+ 46us) write_op.cc:276] Timestamp: P: 1751032523859707 usec, L: 0
0627 13:55:35.137572 (+ 156us) write_op.cc:270] Start()
0627 13:55:35.137591 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523864581 usec, L: 0
0627 13:55:35.142693 (+ 5102us) write_op.cc:270] Start()
0627 13:55:35.142728 (+ 35us) write_op.cc:276] Timestamp: P: 1751032523892024 usec, L: 0
0627 13:55:35.145964 (+ 3236us) write_op.cc:270] Start()
0627 13:55:35.145984 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523909066 usec, L: 0
0627 13:55:35.147055 (+ 1071us) write_op.cc:270] Start()
0627 13:55:35.147079 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523913070 usec, L: 0
0627 13:55:35.158819 (+ 11740us) write_op.cc:270] Start()
0627 13:55:35.158845 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523916482 usec, L: 0
0627 13:55:35.159687 (+ 842us) write_op.cc:270] Start()
0627 13:55:35.159709 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523939184 usec, L: 0
0627 13:55:35.163966 (+ 4257us) write_op.cc:270] Start()
0627 13:55:35.163991 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523958899 usec, L: 0
0627 13:55:35.166242 (+ 2251us) write_op.cc:270] Start()
0627 13:55:35.166260 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523969095 usec, L: 0
0627 13:55:35.166441 (+ 181us) write_op.cc:270] Start()
0627 13:55:35.166462 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523972847 usec, L: 0
0627 13:55:35.166801 (+ 339us) write_op.cc:270] Start()
0627 13:55:35.166818 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524014477 usec, L: 0
0627 13:55:35.166941 (+ 123us) write_op.cc:270] Start()
0627 13:55:35.166955 (+ 14us) write_op.cc:276] Timestamp: P: 1751032524024745 usec, L: 0
0627 13:55:35.167110 (+ 155us) write_op.cc:270] Start()
0627 13:55:35.167129 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524042235 usec, L: 0
0627 13:55:35.169405 (+ 2276us) write_op.cc:270] Start()
0627 13:55:35.169426 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524056673 usec, L: 0
0627 13:55:35.173128 (+ 3702us) write_op.cc:270] Start()
0627 13:55:35.173152 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524071212 usec, L: 0
0627 13:55:35.173336 (+ 184us) write_op.cc:270] Start()
0627 13:55:35.173357 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524080807 usec, L: 0
0627 13:55:35.173502 (+ 145us) write_op.cc:270] Start()
0627 13:55:35.173565 (+ 63us) write_op.cc:276] Timestamp: P: 1751032524081444 usec, L: 0
0627 13:55:35.192158 (+ 18593us) write_op.cc:270] Start()
0627 13:55:35.192188 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524111736 usec, L: 0
0627 13:55:35.196277 (+ 4089us) write_op.cc:270] Start()
0627 13:55:35.196303 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524137718 usec, L: 0
0627 13:55:35.204609 (+ 8306us) write_op.cc:270] Start()
0627 13:55:35.204640 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524138429 usec, L: 0
0627 13:55:35.207527 (+ 2887us) write_op.cc:270] Start()
0627 13:55:35.207555 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524156270 usec, L: 0
0627 13:55:35.207711 (+ 156us) write_op.cc:270] Start()
0627 13:55:35.207729 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524160201 usec, L: 0
0627 13:55:35.207891 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.207913 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524181032 usec, L: 0
0627 13:55:35.208078 (+ 165us) write_op.cc:270] Start()
0627 13:55:35.208099 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524188260 usec, L: 0
0627 13:55:35.208236 (+ 137us) write_op.cc:270] Start()
0627 13:55:35.208255 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524204940 usec, L: 0
0627 13:55:35.211476 (+ 3221us) write_op.cc:270] Start()
0627 13:55:35.211509 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524228961 usec, L: 0
0627 13:55:35.214706 (+ 3197us) write_op.cc:270] Start()
0627 13:55:35.214731 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524241522 usec, L: 0
0627 13:55:35.214893 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.214913 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524244488 usec, L: 0
0627 13:55:35.230627 (+ 15714us) write_op.cc:270] Start()
0627 13:55:35.230661 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524260049 usec, L: 0
0627 13:55:35.231612 (+ 951us) write_op.cc:270] Start()
0627 13:55:35.231638 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524276513 usec, L: 0
0627 13:55:35.255225 (+ 23587us) write_op.cc:270] Start()
0627 13:55:35.255250 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524285502 usec, L: 0
0627 13:55:35.255427 (+ 177us) write_op.cc:270] Start()
0627 13:55:35.255449 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524298272 usec, L: 0
0627 13:55:35.255631 (+ 182us) write_op.cc:270] Start()
0627 13:55:35.258853 (+ 3222us) write_op.cc:276] Timestamp: P: 1751032524311263 usec, L: 0
0627 13:55:35.259081 (+ 228us) write_op.cc:270] Start()
0627 13:55:35.259105 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524318641 usec, L: 0
0627 13:55:35.259834 (+ 729us) write_op.cc:270] Start()
0627 13:55:35.259860 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524345048 usec, L: 0
0627 13:55:35.260052 (+ 192us) write_op.cc:270] Start()
0627 13:55:35.260075 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524350941 usec, L: 0
0627 13:55:35.260253 (+ 178us) write_op.cc:270] Start()
0627 13:55:35.260274 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524370569 usec, L: 0
0627 13:55:35.260446 (+ 172us) write_op.cc:270] Start()
0627 13:55:35.260469 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524373029 usec, L: 0
0627 13:55:35.260626 (+ 157us) write_op.cc:270] Start()
0627 13:55:35.260647 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524390645 usec, L: 0
0627 13:55:35.269198 (+ 8551us) write_op.cc:270] Start()
0627 13:55:35.269220 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524404069 usec, L: 0
0627 13:55:35.269373 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.269393 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524425557 usec, L: 0
0627 13:55:35.279807 (+ 10414us) write_op.cc:270] Start()
0627 13:55:35.279833 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524431206 usec, L: 0
0627 13:55:35.290661 (+ 10828us) write_op.cc:270] Start()
0627 13:55:35.290692 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524457238 usec, L: 0
0627 13:55:35.291079 (+ 387us) write_op.cc:270] Start()
0627 13:55:35.291103 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524463311 usec, L: 0
0627 13:55:35.291274 (+ 171us) write_op.cc:270] Start()
0627 13:55:35.291295 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524478939 usec, L: 0
0627 13:55:35.291479 (+ 184us) write_op.cc:270] Start()
0627 13:55:35.291499 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524481477 usec, L: 0
0627 13:55:35.291654 (+ 155us) write_op.cc:270] Start()
0627 13:55:35.291675 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524522647 usec, L: 0
0627 13:55:35.291834 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.291855 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524523565 usec, L: 0
0627 13:55:35.292016 (+ 161us) write_op.cc:270] Start()
0627 13:55:35.292040 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524524404 usec, L: 0
0627 13:55:35.292175 (+ 135us) write_op.cc:270] Start()
0627 13:55:35.292196 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524560874 usec, L: 0
0627 13:55:35.292346 (+ 150us) write_op.cc:270] Start()
0627 13:55:35.292367 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524575836 usec, L: 0
0627 13:55:35.292511 (+ 144us) write_op.cc:270] Start()
0627 13:55:35.292533 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524585025 usec, L: 0
0627 13:55:35.292685 (+ 152us) write_op.cc:270] Start()
0627 13:55:35.292707 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524609477 usec, L: 0
0627 13:55:35.292868 (+ 161us) write_op.cc:270] Start()
0627 13:55:35.292888 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524618139 usec, L: 0
0627 13:55:35.293029 (+ 141us) write_op.cc:270] Start()
0627 13:55:35.293048 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524655775 usec, L: 0
0627 13:55:35.293199 (+ 151us) write_op.cc:270] Start()
0627 13:55:35.293220 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524661841 usec, L: 0
0627 13:55:35.293377 (+ 157us) write_op.cc:270] Start()
0627 13:55:35.293401 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524662613 usec, L: 0
0627 13:55:35.293549 (+ 148us) write_op.cc:270] Start()
0627 13:55:35.293569 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524703280 usec, L: 0
0627 13:55:35.293728 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.293748 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524709707 usec, L: 0
0627 13:55:35.293902 (+ 154us) write_op.cc:270] Start()
0627 13:55:35.293921 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524712454 usec, L: 0
0627 13:55:35.295163 (+ 1242us) write_op.cc:270] Start()
0627 13:55:35.295183 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524727404 usec, L: 0
0627 13:55:35.297490 (+ 2307us) write_op.cc:270] Start()
0627 13:55:35.297519 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524744182 usec, L: 0
0627 13:55:35.301412 (+ 3893us) write_op.cc:270] Start()
0627 13:55:35.301442 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524761743 usec, L: 0
0627 13:55:35.301636 (+ 194us) write_op.cc:270] Start()
0627 13:55:35.301662 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524764704 usec, L: 0
0627 13:55:35.321239 (+ 19577us) write_op.cc:270] Start()
0627 13:55:35.321272 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524773278 usec, L: 0
0627 13:55:35.331952 (+ 10680us) write_op.cc:270] Start()
0627 13:55:35.331981 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524802929 usec, L: 0
0627 13:55:35.338493 (+ 6512us) write_op.cc:270] Start()
0627 13:55:35.338524 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524817580 usec, L: 0
0627 13:55:35.341282 (+ 2758us) write_op.cc:270] Start()
0627 13:55:35.341306 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524823098 usec, L: 0
0627 13:55:35.352930 (+ 11624us) write_op.cc:270] Start()
0627 13:55:35.354951 (+ 2021us) write_op.cc:276] Timestamp: P: 1751032524823688 usec, L: 0
0627 13:55:35.356209 (+ 1258us) write_op.cc:270] Start()
0627 13:55:35.360689 (+ 4480us) write_op.cc:276] Timestamp: P: 1751032524854748 usec, L: 0
0627 13:55:35.378672 (+ 17983us) write_op.cc:270] Start()
0627 13:55:35.378702 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524868183 usec, L: 0
0627 13:55:35.383448 (+ 4746us) write_op.cc:270] Start()
0627 13:55:35.383482 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524877382 usec, L: 0
0627 13:55:35.391436 (+ 7954us) write_op.cc:270] Start()
0627 13:55:35.391474 (+ 38us) write_op.cc:276] Timestamp: P: 1751032524878711 usec, L: 0
0627 13:55:35.392988 (+ 1514us) write_op.cc:270] Start()
0627 13:55:35.393015 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524905847 usec, L: 0
0627 13:55:35.393218 (+ 203us) write_op.cc:270] Start()
0627 13:55:35.393240 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524914653 usec, L: 0
0627 13:55:35.393418 (+ 178us) write_op.cc:270] Start()
0627 13:55:35.393439 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524931384 usec, L: 0
0627 13:55:35.393591 (+ 152us) write_op.cc:270] Start()
0627 13:55:35.393610 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524943964 usec, L: 0
0627 13:55:35.393757 (+ 147us) write_op.cc:270] Start()
0627 13:55:35.393779 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524951363 usec, L: 0
0627 13:55:35.398578 (+ 4799us) write_op.cc:270] Start()
0627 13:55:35.398607 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524977457 usec, L: 0
0627 13:55:35.398810 (+ 203us) write_op.cc:270] Start()
0627 13:55:35.398837 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524999064 usec, L: 0
0627 13:55:35.399013 (+ 176us) write_op.cc:270] Start()
0627 13:55:35.399035 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525008481 usec, L: 0
0627 13:55:35.399187 (+ 152us) write_op.cc:270] Start()
0627 13:55:35.399207 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525024743 usec, L: 0
0627 13:55:35.399385 (+ 178us) write_op.cc:270] Start()
0627 13:55:35.399407 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525036192 usec, L: 0
0627 13:55:35.413286 (+ 13879us) write_op.cc:270] Start()
0627 13:55:35.413309 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525036901 usec, L: 0
0627 13:55:35.420210 (+ 6901us) write_op.cc:270] Start()
0627 13:55:35.420237 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525059677 usec, L: 0
0627 13:55:35.420428 (+ 191us) write_op.cc:270] Start()
0627 13:55:35.420452 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525092270 usec, L: 0
0627 13:55:35.423600 (+ 3148us) write_op.cc:270] Start()
0627 13:55:35.423624 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525100746 usec, L: 0
0627 13:55:35.425841 (+ 2217us) write_op.cc:270] Start()
0627 13:55:35.425860 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525105960 usec, L: 0
0627 13:55:35.431389 (+ 5529us) write_op.cc:270] Start()
0627 13:55:35.431405 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525122845 usec, L: 0
0627 13:55:35.438409 (+ 7004us) write_op.cc:270] Start()
0627 13:55:35.438431 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525155962 usec, L: 0
0627 13:55:35.441297 (+ 2866us) write_op.cc:270] Start()
0627 13:55:35.441323 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525161951 usec, L: 0
0627 13:55:35.441496 (+ 173us) write_op.cc:270] Start()
0627 13:55:35.441516 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525168624 usec, L: 0
0627 13:55:35.441695 (+ 179us) write_op.cc:270] Start()
0627 13:55:35.441715 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525183120 usec, L: 0
0627 13:55:35.441850 (+ 135us) write_op.cc:270] Start()
0627 13:55:35.441869 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525207556 usec, L: 0
0627 13:55:35.451417 (+ 9548us) write_op.cc:270] Start()
0627 13:55:35.451441 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525214390 usec, L: 0
0627 13:55:35.452666 (+ 1225us) write_op.cc:270] Start()
0627 13:55:35.452695 (+ 29us) write_op.cc:276] Timestamp: P: 1751032525238056 usec, L: 0
0627 13:55:35.458568 (+ 5873us) write_op.cc:270] Start()
0627 13:55:35.458596 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525239726 usec, L: 0
0627 13:55:35.461066 (+ 2470us) write_op.cc:270] Start()
0627 13:55:35.461090 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525256136 usec, L: 0
0627 13:55:35.470095 (+ 9005us) write_op.cc:270] Start()
0627 13:55:35.470120 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525284554 usec, L: 0
0627 13:55:35.472218 (+ 2098us) write_op.cc:270] Start()
0627 13:55:35.472246 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525291608 usec, L: 0
0627 13:55:35.489538 (+ 17292us) write_op.cc:270] Start()
0627 13:55:35.489570 (+ 32us) write_op.cc:276] Timestamp: P: 1751032525294521 usec, L: 0
0627 13:55:35.489781 (+ 211us) write_op.cc:270] Start()
0627 13:55:35.489806 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525295922 usec, L: 0
0627 13:55:35.489983 (+ 177us) write_op.cc:270] Start()
0627 13:55:35.490006 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525335002 usec, L: 0
0627 13:55:35.490190 (+ 184us) write_op.cc:270] Start()
0627 13:55:35.490212 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525346495 usec, L: 0
0627 13:55:35.495949 (+ 5737us) write_op.cc:270] Start()
0627 13:55:35.496051 (+ 102us) write_op.cc:276] Timestamp: P: 1751032525348788 usec, L: 0
0627 13:55:35.496344 (+ 293us) write_op.cc:270] Start()
0627 13:55:35.496370 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525349589 usec, L: 0
0627 13:55:35.496545 (+ 175us) write_op.cc:270] Start()
0627 13:55:35.496566 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525379136 usec, L: 0
0627 13:55:35.496725 (+ 159us) write_op.cc:270] Start()
0627 13:55:35.496746 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525390579 usec, L: 0
0627 13:55:35.496906 (+ 160us) write_op.cc:270] Start()
0627 13:55:35.496926 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525399545 usec, L: 0
0627 13:55:35.497064 (+ 138us) write_op.cc:270] Start()
0627 13:55:35.497086 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525409824 usec, L: 0
0627 13:55:35.508035 (+ 10949us) write_op.cc:270] Start()
0627 13:55:35.508074 (+ 39us) write_op.cc:276] Timestamp: P: 1751032525433281 usec, L: 0
0627 13:55:35.508346 (+ 272us) write_op.cc:270] Start()
0627 13:55:35.508373 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525436112 usec, L: 0
0627 13:55:35.508563 (+ 190us) write_op.cc:270] Start()
0627 13:55:35.508588 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525439478 usec, L: 0
0627 13:55:35.508768 (+ 180us) write_op.cc:270] Start()
0627 13:55:35.508791 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525477023 usec, L: 0
0627 13:55:35.508943 (+ 152us) write_op.cc:270] Start()
0627 13:55:35.508963 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525490423 usec, L: 0
0627 13:55:35.509108 (+ 145us) write_op.cc:270] Start()
0627 13:55:35.509127 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525501564 usec, L: 0
0627 13:55:35.511269 (+ 2142us) write_op.cc:270] Start()
0627 13:55:35.511295 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525506599 usec, L: 0
0627 13:55:35.511448 (+ 153us) write_op.cc:270] Start()
0627 13:55:35.511800 (+ 352us) write_op.cc:276] Timestamp: P: 1751032525522157 usec, L: 0
0627 13:55:35.511976 (+ 176us) write_op.cc:270] Start()
0627 13:55:35.511997 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525586835 usec, L: 0
0627 13:55:35.512145 (+ 148us) write_op.cc:270] Start()
0627 13:55:35.512165 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525588021 usec, L: 0
0627 13:55:35.512308 (+ 143us) write_op.cc:270] Start()
0627 13:55:35.512327 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525589563 usec, L: 0
0627 13:55:35.530334 (+ 18007us) write_op.cc:270] Start()
0627 13:55:35.530360 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525590163 usec, L: 0
0627 13:55:35.535297 (+ 4937us) write_op.cc:270] Start()
0627 13:55:35.535315 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525625864 usec, L: 0
0627 13:55:35.552098 (+ 16783us) write_op.cc:270] Start()
0627 13:55:35.552126 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525639876 usec, L: 0
0627 13:55:35.553741 (+ 1615us) write_op.cc:270] Start()
0627 13:55:35.553761 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525642761 usec, L: 0
0627 13:55:35.554626 (+ 865us) write_op.cc:270] Start()
0627 13:55:35.554652 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525645278 usec, L: 0
0627 13:55:35.560181 (+ 5529us) write_op.cc:270] Start()
0627 13:55:35.560206 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525685983 usec, L: 0
0627 13:55:35.561337 (+ 1131us) write_op.cc:270] Start()
0627 13:55:35.561354 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525701204 usec, L: 0
0627 13:55:35.563071 (+ 1717us) write_op.cc:270] Start()
0627 13:55:35.563097 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525704545 usec, L: 0
0627 13:55:35.563285 (+ 188us) write_op.cc:270] Start()
0627 13:55:35.563311 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525718676 usec, L: 0
0627 13:55:35.563460 (+ 149us) write_op.cc:270] Start()
0627 13:55:35.563482 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525740094 usec, L: 0
0627 13:55:35.563609 (+ 127us) write_op.cc:270] Start()
0627 13:55:35.563644 (+ 35us) write_op.cc:276] Timestamp: P: 1751032525764276 usec, L: 0
0627 13:55:35.563799 (+ 155us) write_op.cc:270] Start()
0627 13:55:35.563818 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525772720 usec, L: 0
0627 13:55:35.563980 (+ 162us) write_op.cc:270] Start()
0627 13:55:35.564000 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525781756 usec, L: 0
0627 13:55:35.564177 (+ 177us) write_op.cc:270] Start()
0627 13:55:35.564198 (+ 21us) write_op.cc:276] Timestamp: P: 17510325
I20250627 13:55:40.108652 9764 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=160 overwritten=0 applied=158 ignored=0} inserts{seen=1265 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250627 13:55:40.114233 9764 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:40.121886 9764 ts_tablet_manager.cc:1397] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 5.557s user 0.296s sys 0.080s
I20250627 13:55:40.152151 9764 raft_consensus.cc:357] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:40.153311 9764 raft_consensus.cc:738] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:40.153844 9764 consensus_queue.cc:260] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 158, Last appended: 1.160, Last appended by leader: 160, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:40.232388 9764 ts_tablet_manager.cc:1428] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.102s user 0.008s sys 0.033s
I20250627 13:55:40.251197 9764 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap starting.
I20250627 13:55:40.550407 9919 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:36.656607 9829 rpcz_store.cc:276] 0627 13:55:34.344733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:34.350125 (+ 5392us) service_pool.cc:225] Handling call
0627 13:55:34.486404 (+136279us) raft_consensus.cc:1469] Updating replica for 603 ops
0627 13:55:34.619411 (+133007us) raft_consensus.cc:1532] Early marking committed up to index 0
0627 13:55:34.619416 (+ 5us) raft_consensus.cc:1537] Triggering prepare for 602 ops
0627 13:55:34.619572 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.620620 (+ 1048us) write_op.cc:276] Timestamp: P: 1751032522406754 usec, L: 0
0627 13:55:34.620836 (+ 216us) write_op.cc:270] Start()
0627 13:55:34.620862 (+ 26us) write_op.cc:276] Timestamp: P: 1751032522414702 usec, L: 0
0627 13:55:34.621016 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.621043 (+ 27us) write_op.cc:276] Timestamp: P: 1751032522418402 usec, L: 0
0627 13:55:34.621221 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.621242 (+ 21us) write_op.cc:276] Timestamp: P: 1751032522649314 usec, L: 0
0627 13:55:34.621389 (+ 147us) write_op.cc:270] Start()
0627 13:55:34.621408 (+ 19us) write_op.cc:276] Timestamp: P: 1751032522662995 usec, L: 0
0627 13:55:34.621570 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.621590 (+ 20us) write_op.cc:276] Timestamp: P: 1751032522989185 usec, L: 0
0627 13:55:34.621753 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.621801 (+ 48us) write_op.cc:276] Timestamp: P: 1751032523018900 usec, L: 0
0627 13:55:34.621955 (+ 154us) write_op.cc:270] Start()
0627 13:55:34.621976 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523022949 usec, L: 0
0627 13:55:34.622147 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.622177 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523047496 usec, L: 0
0627 13:55:34.622350 (+ 173us) write_op.cc:270] Start()
0627 13:55:34.622371 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523075321 usec, L: 0
0627 13:55:34.622522 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.622543 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523081119 usec, L: 0
0627 13:55:34.622700 (+ 157us) write_op.cc:270] Start()
0627 13:55:34.622720 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523108195 usec, L: 0
0627 13:55:34.622863 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.622882 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523110729 usec, L: 0
0627 13:55:34.626733 (+ 3851us) write_op.cc:270] Start()
0627 13:55:34.626761 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523111889 usec, L: 0
0627 13:55:34.626994 (+ 233us) write_op.cc:270] Start()
0627 13:55:34.627020 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523113156 usec, L: 0
0627 13:55:34.627217 (+ 197us) write_op.cc:270] Start()
0627 13:55:34.627237 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523148343 usec, L: 0
0627 13:55:34.627368 (+ 131us) write_op.cc:270] Start()
0627 13:55:34.627394 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523178794 usec, L: 0
0627 13:55:34.627524 (+ 130us) write_op.cc:270] Start()
0627 13:55:34.627542 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523181210 usec, L: 0
0627 13:55:34.627668 (+ 126us) write_op.cc:270] Start()
0627 13:55:34.627683 (+ 15us) write_op.cc:276] Timestamp: P: 1751032523184464 usec, L: 0
0627 13:55:34.627807 (+ 124us) write_op.cc:270] Start()
0627 13:55:34.627824 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523206438 usec, L: 0
0627 13:55:34.627939 (+ 115us) write_op.cc:270] Start()
0627 13:55:34.627954 (+ 15us) write_op.cc:276] Timestamp: P: 1751032523240800 usec, L: 0
0627 13:55:34.628064 (+ 110us) write_op.cc:270] Start()
0627 13:55:34.628080 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523246610 usec, L: 0
0627 13:55:34.628209 (+ 129us) write_op.cc:270] Start()
0627 13:55:34.628225 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523250979 usec, L: 0
0627 13:55:34.628347 (+ 122us) write_op.cc:270] Start()
0627 13:55:34.628364 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523276846 usec, L: 0
0627 13:55:34.628507 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.628525 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523285664 usec, L: 0
0627 13:55:34.628685 (+ 160us) write_op.cc:270] Start()
0627 13:55:34.628702 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523293024 usec, L: 0
0627 13:55:34.628854 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.628875 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523321726 usec, L: 0
0627 13:55:34.629026 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.629047 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523327607 usec, L: 0
0627 13:55:34.629202 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.629220 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523328767 usec, L: 0
0627 13:55:34.629355 (+ 135us) write_op.cc:270] Start()
0627 13:55:34.629371 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523345803 usec, L: 0
0627 13:55:34.629485 (+ 114us) write_op.cc:270] Start()
0627 13:55:34.629501 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523364829 usec, L: 0
0627 13:55:34.629624 (+ 123us) write_op.cc:270] Start()
0627 13:55:34.629639 (+ 15us) write_op.cc:276] Timestamp: P: 1751032523381188 usec, L: 0
0627 13:55:34.629821 (+ 182us) write_op.cc:270] Start()
0627 13:55:34.629839 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523387996 usec, L: 0
0627 13:55:34.629956 (+ 117us) write_op.cc:270] Start()
0627 13:55:34.629972 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523396425 usec, L: 0
0627 13:55:34.643355 (+ 13383us) write_op.cc:270] Start()
0627 13:55:34.643381 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523428030 usec, L: 0
0627 13:55:34.643533 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.643553 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523437735 usec, L: 0
0627 13:55:34.643696 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.643763 (+ 67us) write_op.cc:276] Timestamp: P: 1751032523455347 usec, L: 0
0627 13:55:34.643904 (+ 141us) write_op.cc:270] Start()
0627 13:55:34.643921 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523456711 usec, L: 0
0627 13:55:34.644041 (+ 120us) write_op.cc:270] Start()
0627 13:55:34.644059 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523464720 usec, L: 0
0627 13:55:34.667637 (+ 23578us) write_op.cc:270] Start()
0627 13:55:34.667676 (+ 39us) write_op.cc:276] Timestamp: P: 1751032523493218 usec, L: 0
0627 13:55:34.667877 (+ 201us) write_op.cc:270] Start()
0627 13:55:34.667898 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523515812 usec, L: 0
0627 13:55:34.668044 (+ 146us) write_op.cc:270] Start()
0627 13:55:34.668064 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523521336 usec, L: 0
0627 13:55:34.668224 (+ 160us) write_op.cc:270] Start()
0627 13:55:34.668245 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523532143 usec, L: 0
0627 13:55:34.668377 (+ 132us) write_op.cc:270] Start()
0627 13:55:34.668396 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523546610 usec, L: 0
0627 13:55:34.668540 (+ 144us) write_op.cc:270] Start()
0627 13:55:34.668560 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523574137 usec, L: 0
0627 13:55:34.668712 (+ 152us) write_op.cc:270] Start()
0627 13:55:34.668732 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523579144 usec, L: 0
0627 13:55:34.673363 (+ 4631us) write_op.cc:270] Start()
0627 13:55:34.673390 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523608195 usec, L: 0
0627 13:55:34.685304 (+ 11914us) write_op.cc:270] Start()
0627 13:55:34.685337 (+ 33us) write_op.cc:276] Timestamp: P: 1751032523608841 usec, L: 0
0627 13:55:34.685589 (+ 252us) write_op.cc:270] Start()
0627 13:55:34.685622 (+ 33us) write_op.cc:276] Timestamp: P: 1751032523624394 usec, L: 0
0627 13:55:34.697826 (+ 12204us) write_op.cc:270] Start()
0627 13:55:34.697857 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523629206 usec, L: 0
0627 13:55:34.702089 (+ 4232us) write_op.cc:270] Start()
0627 13:55:34.702121 (+ 32us) write_op.cc:276] Timestamp: P: 1751032523662311 usec, L: 0
0627 13:55:34.702337 (+ 216us) write_op.cc:270] Start()
0627 13:55:34.702363 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523666394 usec, L: 0
0627 13:55:34.702527 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.702549 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523680694 usec, L: 0
0627 13:55:34.702689 (+ 140us) write_op.cc:270] Start()
0627 13:55:34.702712 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523717627 usec, L: 0
0627 13:55:34.702855 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.702874 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523718681 usec, L: 0
0627 13:55:34.703020 (+ 146us) write_op.cc:270] Start()
0627 13:55:34.703039 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523726978 usec, L: 0
0627 13:55:34.703194 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.703215 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523741203 usec, L: 0
0627 13:55:34.703358 (+ 143us) write_op.cc:270] Start()
0627 13:55:34.703378 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523770215 usec, L: 0
0627 13:55:34.704155 (+ 777us) write_op.cc:270] Start()
0627 13:55:34.704181 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523773630 usec, L: 0
0627 13:55:34.705328 (+ 1147us) write_op.cc:270] Start()
0627 13:55:34.705395 (+ 67us) write_op.cc:276] Timestamp: P: 1751032523774871 usec, L: 0
0627 13:55:34.705622 (+ 227us) write_op.cc:270] Start()
0627 13:55:34.705650 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523790656 usec, L: 0
0627 13:55:34.705830 (+ 180us) write_op.cc:270] Start()
0627 13:55:34.705856 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523812435 usec, L: 0
0627 13:55:34.706062 (+ 206us) write_op.cc:270] Start()
0627 13:55:34.706089 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523822692 usec, L: 0
0627 13:55:34.706303 (+ 214us) write_op.cc:270] Start()
0627 13:55:34.706328 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523837427 usec, L: 0
0627 13:55:34.706579 (+ 251us) write_op.cc:270] Start()
0627 13:55:34.706607 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523856781 usec, L: 0
0627 13:55:34.706786 (+ 179us) write_op.cc:270] Start()
0627 13:55:34.706810 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523859682 usec, L: 0
0627 13:55:34.706996 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.707021 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523865256 usec, L: 0
0627 13:55:34.707335 (+ 314us) write_op.cc:270] Start()
0627 13:55:34.707364 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523892714 usec, L: 0
0627 13:55:34.734732 (+ 27368us) write_op.cc:270] Start()
0627 13:55:34.734766 (+ 34us) write_op.cc:276] Timestamp: P: 1751032523908815 usec, L: 0
0627 13:55:34.740499 (+ 5733us) write_op.cc:270] Start()
0627 13:55:34.740542 (+ 43us) write_op.cc:276] Timestamp: P: 1751032523909655 usec, L: 0
0627 13:55:34.742477 (+ 1935us) write_op.cc:270] Start()
0627 13:55:34.742518 (+ 41us) write_op.cc:276] Timestamp: P: 1751032523918647 usec, L: 0
0627 13:55:34.744285 (+ 1767us) write_op.cc:270] Start()
0627 13:55:34.744323 (+ 38us) write_op.cc:276] Timestamp: P: 1751032523939325 usec, L: 0
0627 13:55:34.744603 (+ 280us) write_op.cc:270] Start()
0627 13:55:34.744632 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523960178 usec, L: 0
0627 13:55:34.757578 (+ 12946us) write_op.cc:270] Start()
0627 13:55:34.757628 (+ 50us) write_op.cc:276] Timestamp: P: 1751032523971441 usec, L: 0
0627 13:55:34.757890 (+ 262us) write_op.cc:270] Start()
0627 13:55:34.757912 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523987444 usec, L: 0
0627 13:55:34.758774 (+ 862us) write_op.cc:270] Start()
0627 13:55:34.758814 (+ 40us) write_op.cc:276] Timestamp: P: 1751032524004457 usec, L: 0
0627 13:55:34.764008 (+ 5194us) write_op.cc:270] Start()
0627 13:55:34.764034 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524021756 usec, L: 0
0627 13:55:34.764234 (+ 200us) write_op.cc:270] Start()
0627 13:55:34.764256 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524038624 usec, L: 0
0627 13:55:34.764430 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.764507 (+ 77us) write_op.cc:276] Timestamp: P: 1751032524050604 usec, L: 0
0627 13:55:34.764709 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.764730 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524073601 usec, L: 0
0627 13:55:34.764874 (+ 144us) write_op.cc:270] Start()
0627 13:55:34.764894 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524082973 usec, L: 0
0627 13:55:34.765044 (+ 150us) write_op.cc:270] Start()
0627 13:55:34.765063 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524099644 usec, L: 0
0627 13:55:34.765221 (+ 158us) write_op.cc:270] Start()
0627 13:55:34.765250 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524112067 usec, L: 0
0627 13:55:34.765479 (+ 229us) write_op.cc:270] Start()
0627 13:55:34.765506 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524139113 usec, L: 0
0627 13:55:34.765676 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.765700 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524141054 usec, L: 0
0627 13:55:34.773817 (+ 8117us) write_op.cc:270] Start()
0627 13:55:34.773855 (+ 38us) write_op.cc:276] Timestamp: P: 1751032524150957 usec, L: 0
0627 13:55:34.774123 (+ 268us) write_op.cc:270] Start()
0627 13:55:34.774152 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524156908 usec, L: 0
0627 13:55:34.776604 (+ 2452us) write_op.cc:270] Start()
0627 13:55:34.776644 (+ 40us) write_op.cc:276] Timestamp: P: 1751032524180403 usec, L: 0
0627 13:55:34.783172 (+ 6528us) write_op.cc:270] Start()
0627 13:55:34.783196 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524187640 usec, L: 0
0627 13:55:34.783851 (+ 655us) write_op.cc:270] Start()
0627 13:55:34.783872 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524201841 usec, L: 0
0627 13:55:34.784034 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.784054 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524225292 usec, L: 0
0627 13:55:34.784209 (+ 155us) write_op.cc:270] Start()
0627 13:55:34.784227 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524237335 usec, L: 0
0627 13:55:34.784725 (+ 498us) write_op.cc:270] Start()
0627 13:55:34.784749 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524244687 usec, L: 0
0627 13:55:34.784917 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.784940 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524254265 usec, L: 0
0627 13:55:34.788510 (+ 3570us) write_op.cc:270] Start()
0627 13:55:34.788537 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524280368 usec, L: 0
0627 13:55:34.788723 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.788744 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524283604 usec, L: 0
0627 13:55:34.788868 (+ 124us) write_op.cc:270] Start()
0627 13:55:34.788885 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524298984 usec, L: 0
0627 13:55:34.789022 (+ 137us) write_op.cc:270] Start()
0627 13:55:34.789041 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524311902 usec, L: 0
0627 13:55:34.789181 (+ 140us) write_op.cc:270] Start()
0627 13:55:34.789199 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524319271 usec, L: 0
0627 13:55:34.789333 (+ 134us) write_op.cc:270] Start()
0627 13:55:34.789350 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524344840 usec, L: 0
0627 13:55:34.789506 (+ 156us) write_op.cc:270] Start()
0627 13:55:34.789524 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524353184 usec, L: 0
0627 13:55:34.790408 (+ 884us) write_op.cc:270] Start()
0627 13:55:34.790436 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524372032 usec, L: 0
0627 13:55:34.790644 (+ 208us) write_op.cc:270] Start()
0627 13:55:34.790666 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524375831 usec, L: 0
0627 13:55:34.807331 (+ 16665us) write_op.cc:270] Start()
0627 13:55:34.807352 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524398825 usec, L: 0
0627 13:55:34.807913 (+ 561us) write_op.cc:270] Start()
0627 13:55:34.807943 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524403340 usec, L: 0
0627 13:55:34.808131 (+ 188us) write_op.cc:270] Start()
0627 13:55:34.808156 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524425252 usec, L: 0
0627 13:55:34.808345 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.808368 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524430978 usec, L: 0
0627 13:55:34.808530 (+ 162us) write_op.cc:270] Start()
0627 13:55:34.808556 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524455173 usec, L: 0
0627 13:55:34.808717 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.808738 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524460556 usec, L: 0
0627 13:55:34.808886 (+ 148us) write_op.cc:270] Start()
0627 13:55:34.808906 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524482232 usec, L: 0
0627 13:55:34.809087 (+ 181us) write_op.cc:270] Start()
0627 13:55:34.809107 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524483763 usec, L: 0
0627 13:55:34.809297 (+ 190us) write_op.cc:270] Start()
0627 13:55:34.809318 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524519102 usec, L: 0
0627 13:55:34.819864 (+ 10546us) write_op.cc:270] Start()
0627 13:55:34.819886 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524520556 usec, L: 0
0627 13:55:34.820100 (+ 214us) write_op.cc:270] Start()
0627 13:55:34.820123 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524556145 usec, L: 0
0627 13:55:34.820299 (+ 176us) write_op.cc:270] Start()
0627 13:55:34.820321 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524570000 usec, L: 0
0627 13:55:34.820471 (+ 150us) write_op.cc:270] Start()
0627 13:55:34.820495 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524570653 usec, L: 0
0627 13:55:34.820654 (+ 159us) write_op.cc:270] Start()
0627 13:55:34.820674 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524576029 usec, L: 0
0627 13:55:34.820848 (+ 174us) write_op.cc:270] Start()
0627 13:55:34.820870 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524610532 usec, L: 0
0627 13:55:34.821012 (+ 142us) write_op.cc:270] Start()
0627 13:55:34.821033 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524621565 usec, L: 0
0627 13:55:34.821196 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.821217 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524641481 usec, L: 0
0627 13:55:34.821368 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.821389 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524654030 usec, L: 0
0627 13:55:34.821550 (+ 161us) write_op.cc:270] Start()
0627 13:55:34.821570 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524662758 usec, L: 0
0627 13:55:34.821729 (+ 159us) write_op.cc:270] Start()
0627 13:55:34.821749 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524690963 usec, L: 0
0627 13:55:34.821899 (+ 150us) write_op.cc:270] Start()
0627 13:55:34.821919 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524699114 usec, L: 0
0627 13:55:34.822085 (+ 166us) write_op.cc:270] Start()
0627 13:55:34.822105 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524707348 usec, L: 0
0627 13:55:34.822269 (+ 164us) write_op.cc:270] Start()
0627 13:55:34.822291 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524709604 usec, L: 0
0627 13:55:34.822440 (+ 149us) write_op.cc:270] Start()
0627 13:55:34.822462 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524742830 usec, L: 0
0627 13:55:34.822651 (+ 189us) write_op.cc:270] Start()
0627 13:55:34.822671 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524760982 usec, L: 0
0627 13:55:34.822884 (+ 213us) write_op.cc:270] Start()
0627 13:55:34.822905 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524762833 usec, L: 0
0627 13:55:34.823056 (+ 151us) write_op.cc:270] Start()
0627 13:55:34.823079 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524764800 usec, L: 0
0627 13:55:34.823249 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.823292 (+ 43us) write_op.cc:276] Timestamp: P: 1751032524804353 usec, L: 0
0627 13:55:34.823445 (+ 153us) write_op.cc:270] Start()
0627 13:55:34.823466 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524818444 usec, L: 0
0627 13:55:34.823642 (+ 176us) write_op.cc:270] Start()
0627 13:55:34.823662 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524819646 usec, L: 0
0627 13:55:34.823812 (+ 150us) write_op.cc:270] Start()
0627 13:55:34.823832 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524820311 usec, L: 0
0627 13:55:34.823995 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.824016 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524855491 usec, L: 0
0627 13:55:34.824781 (+ 765us) write_op.cc:270] Start()
0627 13:55:34.824808 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524869341 usec, L: 0
0627 13:55:34.865985 (+ 41177us) write_op.cc:270] Start()
0627 13:55:34.866018 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524876805 usec, L: 0
0627 13:55:34.866264 (+ 246us) write_op.cc:270] Start()
0627 13:55:34.866290 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524884520 usec, L: 0
0627 13:55:34.866932 (+ 642us) write_op.cc:270] Start()
0627 13:55:34.866969 (+ 37us) write_op.cc:276] Timestamp: P: 1751032524906357 usec, L: 0
0627 13:55:34.867175 (+ 206us) write_op.cc:270] Start()
0627 13:55:34.867204 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524920161 usec, L: 0
0627 13:55:34.867675 (+ 471us) write_op.cc:270] Start()
0627 13:55:34.867707 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524932571 usec, L: 0
0627 13:55:34.867903 (+ 196us) write_op.cc:270] Start()
0627 13:55:34.867954 (+ 51us) write_op.cc:276] Timestamp: P: 1751032524946133 usec, L: 0
0627 13:55:34.868153 (+ 199us) write_op.cc:270] Start()
0627 13:55:34.868179 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524951583 usec, L: 0
0627 13:55:34.868390 (+ 211us) write_op.cc:270] Start()
0627 13:55:34.868416 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524976730 usec, L: 0
0627 13:55:34.868594 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.868623 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524983385 usec, L: 0
0627 13:55:34.869226 (+ 603us) write_op.cc:270] Start()
0627 13:55:34.869270 (+ 44us) write_op.cc:276] Timestamp: P: 1751032524999266 usec, L: 0
0627 13:55:34.869502 (+ 232us) write_op.cc:270] Start()
0627 13:55:34.869529 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525025452 usec, L: 0
0627 13:55:34.869751 (+ 222us) write_op.cc:270] Start()
0627 13:55:34.869777 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525036201 usec, L: 0
0627 13:55:34.869979 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.870005 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525039675 usec, L: 0
0627 13:55:34.870221 (+ 216us) write_op.cc:270] Start()
0627 13:55:34.870261 (+ 40us) write_op.cc:276] Timestamp: P: 1751032525058647 usec, L: 0
0627 13:55:34.870445 (+ 184us) write_op.cc:270] Start()
0627 13:55:34.870471 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525092695 usec, L: 0
0627 13:55:34.870661 (+ 190us) write_op.cc:270] Start()
0627 13:55:34.870688 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525100039 usec, L: 0
0627 13:55:34.870856 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.870882 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525103936 usec, L: 0
0627 13:55:34.871065 (+ 183us) write_op.cc:270] Start()
0627 13:55:34.871081 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525129446 usec, L: 0
0627 13:55:34.871207 (+ 126us) write_op.cc:270] Start()
0627 13:55:34.871232 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525155227 usec, L: 0
0627 13:55:34.871464 (+ 232us) write_op.cc:270] Start()
0627 13:55:34.871491 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525160375 usec, L: 0
0627 13:55:34.871673 (+ 182us) write_op.cc:270] Start()
0627 13:55:34.871698 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525168724 usec, L: 0
0627 13:55:34.882952 (+ 11254us) write_op.cc:270] Start()
0627 13:55:34.883010 (+ 58us) write_op.cc:276] Timestamp: P: 1751032525182517 usec, L: 0
0627 13:55:34.883204 (+ 194us) write_op.cc:270] Start()
0627 13:55:34.883231 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525207693 usec, L: 0
0627 13:55:34.883456 (+ 225us) write_op.cc:270] Start()
0627 13:55:34.883484 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525215025 usec, L: 0
0627 13:55:34.883660 (+ 176us) write_op.cc:270] Start()
0627 13:55:34.883687 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525220348 usec, L: 0
0627 13:55:34.883857 (+ 170us) write_op.cc:270] Start()
0627 13:55:34.883889 (+ 32us) write_op.cc:276] Timestamp: P: 1751032525238769 usec, L: 0
0627 13:55:34.884057 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.884081 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525253287 usec, L: 0
0627 13:55:34.884305 (+ 224us) write_op.cc:270] Start()
0627 13:55:34.884331 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525285259 usec, L: 0
0627 13:55:34.884510 (+ 179us) write_op.cc:270] Start()
0627 13:55:34.884532 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525290675 usec, L: 0
0627 13:55:34.884709 (+ 177us) write_op.cc:270] Start()
0627 13:55:34.884737 (+ 28us) write_op.cc:276] Timestamp: P: 1751032525291902 usec, L: 0
0627 13:55:34.884994 (+ 257us) write_op.cc:270] Start()
0627 13:55:34.885131 (+ 137us) write_op.cc:276] Timestamp: P: 1751032525305414 usec, L: 0
0627 13:55:34.885392 (+ 261us) write_op.cc:270] Start()
0627 13:55:34.885418 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525326655 usec, L: 0
0627 13:55:34.885715 (+ 297us) write_op.cc:270] Start()
0627 13:55:34.885748 (+ 33us) write_op.cc:276] Timestamp: P: 1751032525338539 usec, L: 0
0627 13:55:34.885971 (+ 223us) write_op.cc:270] Start()
0627 13:55:34.885996 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525347809 usec, L: 0
0627 13:55:34.886190 (+ 194us) write_op.cc:270] Start()
0627 13:55:34.886216 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525349534 usec, L: 0
0627 13:55:34.886418 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.886444 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525379102 usec, L: 0
0627 13:55:34.886649 (+ 205us) write_op.cc:270] Start()
0627 13:55:34.886674 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525389811 usec, L: 0
0627 13:55:34.886852 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.886878 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525410590 usec, L: 0
0627 13:55:34.887120 (+ 242us) write_op.cc:270] Start()
0627 13:55:34.887145 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525411277 usec, L: 0
0627 13:55:34.887321 (+ 176us) write_op.cc:270] Start()
0627 13:55:34.887346 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525433111 usec, L: 0
0627 13:55:34.887511 (+ 165us) write_op.cc:270] Start()
0627 13:55:34.887536 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525433969 usec, L: 0
0627 13:55:34.887730 (+ 194us) write_op.cc:270] Start()
0627 13:55:34.887761 (+ 31us) write_op.cc:276] Timestamp: P: 1751032525444548 usec, L: 0
0627 13:55:34.887997 (+ 236us) write_op.cc:270] Start()
0627 13:55:34.888023 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525477159 usec, L: 0
0627 13:55:34.888201 (+ 178us) write_op.cc:270] Start()
0627 13:55:34.888227 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525501183 usec, L: 0
0627 13:55:34.888408 (+ 181us) write_op.cc:270] Start()
0627 13:55:34.888432 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525502242 usec, L: 0
0627 13:55:34.888603 (+ 171us) write_op.cc:270] Start()
0627 13:55:34.888629 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525507032 usec, L: 0
0627 13:55:34.888815 (+ 186us) write_op.cc:270] Start()
0627 13:55:34.888840 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525523265 usec, L: 0
0627 13:55:34.889003 (+ 163us) write_op.cc:270] Start()
0627 13:55:34.889027 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525587309 usec, L: 0
0627 13:55:34.889227 (+ 200us) write_op.cc:270] Start()
0627 13:55:34.889285 (+ 58us) write_op.cc:276] Timestamp: P: 1751032525588253 usec, L: 0
0627 13:55:34.889499 (+ 214us) write_op.cc:270] Start()
0627 13:55:34.889523 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525589739 usec, L: 0
0627 13:55:34.889732 (+ 209us) write_op.cc:270] Start()
0627 13:55:34.889758 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525594948 usec, L: 0
0627 13:55:34.908274 (+ 18516us) write_op.cc:270] Start()
0627 13:55:34.908300 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525623782 usec, L: 0
0627 13:55:34.908469 (+ 169us) write_op.cc:270] Start()
0627 13:55:34.908493 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525635201 usec, L: 0
0627 13:55:34.908661 (+ 168us) write_op.cc:270] Start()
0627 13:55:34.908693 (+ 32us) write_op.cc:276] Timestamp: P: 1751032525640770 usec, L: 0
0627 13:55:34.908865 (+ 172us) write_op.cc:270] Start()
0627 13:55:34.908888 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525644731 usec, L: 0
0627 13:55:34.909057 (+ 169us) write_op.cc:270] Start()
0627 13:55:34.909082 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525686593 usec, L: 0
0627 13:55:34.909248 (+ 166us) write_op.cc:270] Start()
0627 13:55:34.909284 (+ 36us) write_op.cc:276] Timestamp: P: 1751032525700453 usec, L: 0
0627 13:55:34.909453 (+ 169us) write_op.cc:270] Start()
0627 13:55:34.909479 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525705245 usec, L: 0
0627 13:55:34.909678 (+ 199us) write_op.cc:270] Start()
0627 13:55:34.909708 (+ 30us) write_op.cc:276] Timestamp: P: 1751032525710047 usec, L: 0
0627 13:55:34.909937 (+ 229us) write_op.cc:270] Start()
0627 13:55:34.909963 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525744153 usec, L: 0
0627 13:55:34.910165 (+ 202us) write_op.cc:270] Start()
0627 13:55:34.910191 (+ 26us) write_op.cc:276] Timestamp: P: 17510325
I20250627 13:55:40.550705 9919 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:40.609393 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:40.609037 9919 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:40.776546 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:40.900008 9761 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.140820 9913 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:41.141076 9913 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:41.142027 9913 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:41.143349 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1010 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:41.144949 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1010 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:41.169876 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.170441 9913 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:41.180408 9828 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 4) took 1865 ms. Trace:
I20250627 13:55:41.198422 9827 raft_consensus.cc:3058] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:55:41.458947 9917 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:41.459141 9917 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:41.459920 9830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 28) took 1739 ms. Trace:
I20250627 13:55:41.486927 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:41.490852 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:41.493901 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.496088 9917 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:41.496675 9917 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:41.506319 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250627 13:55:41.599050 9917 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:41.599279 9917 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:41.602398 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1010 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:41.605389 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1010 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:41.618422 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.619176 9917 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:41.626480 9917 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:41.734416 9826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 5) took 1748 ms. Trace:
I20250627 13:55:41.762355 9919 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:41.769654 9919 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:41.773348 9919 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:41.774593 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1050 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
W20250627 13:55:41.835844 9761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 225521us
I20250627 13:55:41.836943 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1050 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:41.837525 9761 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.837877 9919 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:41.181753 9828 rpcz_store.cc:276] 0627 13:55:39.277496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:39.277598 (+ 102us) service_pool.cc:225] Handling call
0627 13:55:39.436906 (+159308us) raft_consensus.cc:1469] Updating replica for 254 ops
0627 13:55:39.715105 (+278199us) raft_consensus.cc:1532] Early marking committed up to index 756
0627 13:55:39.715163 (+ 58us) raft_consensus.cc:1537] Triggering prepare for 254 ops
0627 13:55:39.715294 (+ 131us) write_op.cc:270] Start()
0627 13:55:39.715328 (+ 34us) write_op.cc:276] Timestamp: P: 1751032534225996 usec, L: 0
0627 13:55:39.715501 (+ 173us) write_op.cc:270] Start()
0627 13:55:39.715516 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534242666 usec, L: 0
0627 13:55:39.715640 (+ 124us) write_op.cc:270] Start()
0627 13:55:39.715653 (+ 13us) write_op.cc:276] Timestamp: P: 1751032534243750 usec, L: 0
0627 13:55:39.715734 (+ 81us) write_op.cc:270] Start()
0627 13:55:39.715746 (+ 12us) write_op.cc:276] Timestamp: P: 1751032534280875 usec, L: 0
0627 13:55:39.715832 (+ 86us) write_op.cc:270] Start()
0627 13:55:39.715846 (+ 14us) write_op.cc:276] Timestamp: P: 1751032534345132 usec, L: 0
0627 13:55:39.715984 (+ 138us) write_op.cc:270] Start()
0627 13:55:39.722493 (+ 6509us) write_op.cc:276] Timestamp: P: 1751032534346113 usec, L: 0
0627 13:55:39.727598 (+ 5105us) write_op.cc:270] Start()
0627 13:55:39.743314 (+ 15716us) write_op.cc:276] Timestamp: P: 1751032534349960 usec, L: 0
0627 13:55:39.747479 (+ 4165us) write_op.cc:270] Start()
0627 13:55:39.747512 (+ 33us) write_op.cc:276] Timestamp: P: 1751032534393567 usec, L: 0
0627 13:55:39.765460 (+ 17948us) write_op.cc:270] Start()
0627 13:55:39.765496 (+ 36us) write_op.cc:276] Timestamp: P: 1751032534426088 usec, L: 0
0627 13:55:39.768472 (+ 2976us) write_op.cc:270] Start()
0627 13:55:39.768504 (+ 32us) write_op.cc:276] Timestamp: P: 1751032534431892 usec, L: 0
0627 13:55:39.778458 (+ 9954us) write_op.cc:270] Start()
0627 13:55:39.778492 (+ 34us) write_op.cc:276] Timestamp: P: 1751032534439868 usec, L: 0
0627 13:55:39.778718 (+ 226us) write_op.cc:270] Start()
0627 13:55:39.778745 (+ 27us) write_op.cc:276] Timestamp: P: 1751032534440437 usec, L: 0
0627 13:55:39.802493 (+ 23748us) write_op.cc:270] Start()
0627 13:55:39.802521 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534480274 usec, L: 0
0627 13:55:39.814121 (+ 11600us) write_op.cc:270] Start()
0627 13:55:39.814146 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534510372 usec, L: 0
0627 13:55:39.824298 (+ 10152us) write_op.cc:270] Start()
0627 13:55:39.824329 (+ 31us) write_op.cc:276] Timestamp: P: 1751032534511138 usec, L: 0
0627 13:55:39.824567 (+ 238us) write_op.cc:270] Start()
0627 13:55:39.824592 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534520075 usec, L: 0
0627 13:55:39.824788 (+ 196us) write_op.cc:270] Start()
0627 13:55:39.824810 (+ 22us) write_op.cc:276] Timestamp: P: 1751032534568477 usec, L: 0
0627 13:55:39.824960 (+ 150us) write_op.cc:270] Start()
0627 13:55:39.824983 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534600055 usec, L: 0
0627 13:55:39.825161 (+ 178us) write_op.cc:270] Start()
0627 13:55:39.825184 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534612450 usec, L: 0
0627 13:55:39.825349 (+ 165us) write_op.cc:270] Start()
0627 13:55:39.833385 (+ 8036us) write_op.cc:276] Timestamp: P: 1751032534622885 usec, L: 0
0627 13:55:39.837603 (+ 4218us) write_op.cc:270] Start()
0627 13:55:39.837626 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534651721 usec, L: 0
0627 13:55:39.840919 (+ 3293us) write_op.cc:270] Start()
0627 13:55:39.840947 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534696351 usec, L: 0
0627 13:55:39.845249 (+ 4302us) write_op.cc:270] Start()
0627 13:55:39.845300 (+ 51us) write_op.cc:276] Timestamp: P: 1751032534720991 usec, L: 0
0627 13:55:39.854502 (+ 9202us) write_op.cc:270] Start()
0627 13:55:39.854531 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534734393 usec, L: 0
0627 13:55:39.863279 (+ 8748us) write_op.cc:270] Start()
0627 13:55:39.863313 (+ 34us) write_op.cc:276] Timestamp: P: 1751032534757450 usec, L: 0
0627 13:55:39.864296 (+ 983us) write_op.cc:270] Start()
0627 13:55:39.864324 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534797810 usec, L: 0
0627 13:55:39.864528 (+ 204us) write_op.cc:270] Start()
0627 13:55:39.864548 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534839770 usec, L: 0
0627 13:55:39.864684 (+ 136us) write_op.cc:270] Start()
0627 13:55:39.864702 (+ 18us) write_op.cc:276] Timestamp: P: 1751032534846830 usec, L: 0
0627 13:55:39.864818 (+ 116us) write_op.cc:270] Start()
0627 13:55:39.864834 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534864472 usec, L: 0
0627 13:55:39.864958 (+ 124us) write_op.cc:270] Start()
0627 13:55:39.864973 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534892162 usec, L: 0
0627 13:55:39.865084 (+ 111us) write_op.cc:270] Start()
0627 13:55:39.865099 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534960233 usec, L: 0
0627 13:55:39.865230 (+ 131us) write_op.cc:270] Start()
0627 13:55:39.865246 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534989197 usec, L: 0
0627 13:55:39.865409 (+ 163us) write_op.cc:270] Start()
0627 13:55:39.865427 (+ 18us) write_op.cc:276] Timestamp: P: 1751032534993507 usec, L: 0
0627 13:55:39.865560 (+ 133us) write_op.cc:270] Start()
0627 13:55:39.865575 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534995936 usec, L: 0
0627 13:55:39.865700 (+ 125us) write_op.cc:270] Start()
0627 13:55:39.865716 (+ 16us) write_op.cc:276] Timestamp: P: 1751032535058713 usec, L: 0
0627 13:55:39.865830 (+ 114us) write_op.cc:270] Start()
0627 13:55:39.865846 (+ 16us) write_op.cc:276] Timestamp: P: 1751032535083224 usec, L: 0
0627 13:55:39.865951 (+ 105us) write_op.cc:270] Start()
0627 13:55:39.865966 (+ 15us) write_op.cc:276] Timestamp: P: 1751032535110631 usec, L: 0
0627 13:55:39.866141 (+ 175us) write_op.cc:270] Start()
0627 13:55:39.866160 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535117870 usec, L: 0
0627 13:55:39.866279 (+ 119us) write_op.cc:270] Start()
0627 13:55:39.866298 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535119003 usec, L: 0
0627 13:55:39.873033 (+ 6735us) write_op.cc:270] Start()
0627 13:55:39.873081 (+ 48us) write_op.cc:276] Timestamp: P: 1751032535178382 usec, L: 0
0627 13:55:39.874800 (+ 1719us) write_op.cc:270] Start()
0627 13:55:39.874826 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535188470 usec, L: 0
0627 13:55:39.882583 (+ 7757us) write_op.cc:270] Start()
0627 13:55:39.882603 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535192763 usec, L: 0
0627 13:55:39.886258 (+ 3655us) write_op.cc:270] Start()
0627 13:55:39.886290 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535196417 usec, L: 0
0627 13:55:39.902227 (+ 15937us) write_op.cc:270] Start()
0627 13:55:39.902260 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535221856 usec, L: 0
0627 13:55:39.902523 (+ 263us) write_op.cc:270] Start()
0627 13:55:39.902548 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535252382 usec, L: 0
0627 13:55:39.912550 (+ 10002us) write_op.cc:270] Start()
0627 13:55:39.912580 (+ 30us) write_op.cc:276] Timestamp: P: 1751032535256675 usec, L: 0
0627 13:55:39.915255 (+ 2675us) write_op.cc:270] Start()
0627 13:55:39.915276 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535266790 usec, L: 0
0627 13:55:39.923677 (+ 8401us) write_op.cc:270] Start()
0627 13:55:39.923709 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535290277 usec, L: 0
0627 13:55:39.942683 (+ 18974us) write_op.cc:270] Start()
0627 13:55:39.942715 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535309485 usec, L: 0
0627 13:55:39.948211 (+ 5496us) write_op.cc:270] Start()
0627 13:55:39.948234 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535321768 usec, L: 0
0627 13:55:39.964728 (+ 16494us) write_op.cc:270] Start()
0627 13:55:39.964760 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535333818 usec, L: 0
0627 13:55:39.966346 (+ 1586us) write_op.cc:270] Start()
0627 13:55:39.966375 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535354883 usec, L: 0
0627 13:55:39.966681 (+ 306us) write_op.cc:270] Start()
0627 13:55:39.966705 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535363393 usec, L: 0
0627 13:55:39.980792 (+ 14087us) write_op.cc:270] Start()
0627 13:55:39.980826 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535370339 usec, L: 0
0627 13:55:39.989689 (+ 8863us) write_op.cc:270] Start()
0627 13:55:39.989720 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535403323 usec, L: 0
0627 13:55:40.007344 (+ 17624us) write_op.cc:270] Start()
0627 13:55:40.011441 (+ 4097us) write_op.cc:276] Timestamp: P: 1751032535409875 usec, L: 0
0627 13:55:40.014932 (+ 3491us) write_op.cc:270] Start()
0627 13:55:40.014963 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535438404 usec, L: 0
0627 13:55:40.015202 (+ 239us) write_op.cc:270] Start()
0627 13:55:40.015224 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535440589 usec, L: 0
0627 13:55:40.015831 (+ 607us) write_op.cc:270] Start()
0627 13:55:40.015857 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535494201 usec, L: 0
0627 13:55:40.054563 (+ 38706us) write_op.cc:270] Start()
0627 13:55:40.059235 (+ 4672us) write_op.cc:276] Timestamp: P: 1751032535502878 usec, L: 0
0627 13:55:40.080677 (+ 21442us) write_op.cc:270] Start()
0627 13:55:40.086177 (+ 5500us) write_op.cc:276] Timestamp: P: 1751032535503582 usec, L: 0
0627 13:55:40.089483 (+ 3306us) write_op.cc:270] Start()
0627 13:55:40.089514 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535505685 usec, L: 0
0627 13:55:40.089748 (+ 234us) write_op.cc:270] Start()
0627 13:55:40.089768 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535520003 usec, L: 0
0627 13:55:40.092067 (+ 2299us) write_op.cc:270] Start()
0627 13:55:40.092095 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535583040 usec, L: 0
0627 13:55:40.103865 (+ 11770us) write_op.cc:270] Start()
0627 13:55:40.103895 (+ 30us) write_op.cc:276] Timestamp: P: 1751032535593566 usec, L: 0
0627 13:55:40.107019 (+ 3124us) write_op.cc:270] Start()
0627 13:55:40.107046 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535603241 usec, L: 0
0627 13:55:40.107300 (+ 254us) write_op.cc:270] Start()
0627 13:55:40.107323 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535620062 usec, L: 0
0627 13:55:40.107511 (+ 188us) write_op.cc:270] Start()
0627 13:55:40.107532 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535663310 usec, L: 0
0627 13:55:40.107741 (+ 209us) write_op.cc:270] Start()
0627 13:55:40.107763 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535665870 usec, L: 0
0627 13:55:40.107962 (+ 199us) write_op.cc:270] Start()
0627 13:55:40.107983 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535678758 usec, L: 0
0627 13:55:40.108472 (+ 489us) write_op.cc:270] Start()
0627 13:55:40.108495 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535713251 usec, L: 0
0627 13:55:40.108737 (+ 242us) write_op.cc:270] Start()
0627 13:55:40.108779 (+ 42us) write_op.cc:276] Timestamp: P: 1751032535721743 usec, L: 0
0627 13:55:40.108986 (+ 207us) write_op.cc:270] Start()
0627 13:55:40.109012 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535730752 usec, L: 0
0627 13:55:40.109241 (+ 229us) write_op.cc:270] Start()
0627 13:55:40.109267 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535732207 usec, L: 0
0627 13:55:40.114639 (+ 5372us) write_op.cc:270] Start()
0627 13:55:40.114670 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535787387 usec, L: 0
0627 13:55:40.127734 (+ 13064us) write_op.cc:270] Start()
0627 13:55:40.127768 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535792754 usec, L: 0
0627 13:55:40.128035 (+ 267us) write_op.cc:270] Start()
0627 13:55:40.128058 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535797515 usec, L: 0
0627 13:55:40.128270 (+ 212us) write_op.cc:270] Start()
0627 13:55:40.128290 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535798301 usec, L: 0
0627 13:55:40.128500 (+ 210us) write_op.cc:270] Start()
0627 13:55:40.128520 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535840865 usec, L: 0
0627 13:55:40.128710 (+ 190us) write_op.cc:270] Start()
0627 13:55:40.128733 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535845138 usec, L: 0
0627 13:55:40.130112 (+ 1379us) write_op.cc:270] Start()
0627 13:55:40.130139 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535852746 usec, L: 0
0627 13:55:40.130354 (+ 215us) write_op.cc:270] Start()
0627 13:55:40.130381 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535865397 usec, L: 0
0627 13:55:40.130575 (+ 194us) write_op.cc:270] Start()
0627 13:55:40.130596 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535902737 usec, L: 0
0627 13:55:40.130797 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.130818 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535925684 usec, L: 0
0627 13:55:40.131016 (+ 198us) write_op.cc:270] Start()
0627 13:55:40.131037 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535936451 usec, L: 0
0627 13:55:40.131231 (+ 194us) write_op.cc:270] Start()
0627 13:55:40.131252 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535957538 usec, L: 0
0627 13:55:40.131457 (+ 205us) write_op.cc:270] Start()
0627 13:55:40.131482 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535980935 usec, L: 0
0627 13:55:40.141069 (+ 9587us) write_op.cc:270] Start()
0627 13:55:40.142632 (+ 1563us) write_op.cc:276] Timestamp: P: 1751032536034177 usec, L: 0
0627 13:55:40.149926 (+ 7294us) write_op.cc:270] Start()
0627 13:55:40.149950 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536044846 usec, L: 0
0627 13:55:40.151403 (+ 1453us) write_op.cc:270] Start()
0627 13:55:40.151487 (+ 84us) write_op.cc:276] Timestamp: P: 1751032536063925 usec, L: 0
0627 13:55:40.152683 (+ 1196us) write_op.cc:270] Start()
0627 13:55:40.152709 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536081353 usec, L: 0
0627 13:55:40.154391 (+ 1682us) write_op.cc:270] Start()
0627 13:55:40.154412 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536105236 usec, L: 0
0627 13:55:40.156904 (+ 2492us) write_op.cc:270] Start()
0627 13:55:40.156922 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536147495 usec, L: 0
0627 13:55:40.158860 (+ 1938us) write_op.cc:270] Start()
0627 13:55:40.158885 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536180846 usec, L: 0
0627 13:55:40.159057 (+ 172us) write_op.cc:270] Start()
0627 13:55:40.159078 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536186514 usec, L: 0
0627 13:55:40.183251 (+ 24173us) write_op.cc:270] Start()
0627 13:55:40.183288 (+ 37us) write_op.cc:276] Timestamp: P: 1751032536249554 usec, L: 0
0627 13:55:40.203261 (+ 19973us) write_op.cc:270] Start()
0627 13:55:40.203308 (+ 47us) write_op.cc:276] Timestamp: P: 1751032536255027 usec, L: 0
0627 13:55:40.203611 (+ 303us) write_op.cc:270] Start()
0627 13:55:40.203634 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536262544 usec, L: 0
0627 13:55:40.208729 (+ 5095us) write_op.cc:270] Start()
0627 13:55:40.208774 (+ 45us) write_op.cc:276] Timestamp: P: 1751032536263157 usec, L: 0
0627 13:55:40.209041 (+ 267us) write_op.cc:270] Start()
0627 13:55:40.209070 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536326428 usec, L: 0
0627 13:55:40.209254 (+ 184us) write_op.cc:270] Start()
0627 13:55:40.209279 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536334253 usec, L: 0
0627 13:55:40.211793 (+ 2514us) write_op.cc:270] Start()
0627 13:55:40.211833 (+ 40us) write_op.cc:276] Timestamp: P: 1751032536336446 usec, L: 0
0627 13:55:40.212080 (+ 247us) write_op.cc:270] Start()
0627 13:55:40.212107 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536339680 usec, L: 0
0627 13:55:40.212302 (+ 195us) write_op.cc:270] Start()
0627 13:55:40.212350 (+ 48us) write_op.cc:276] Timestamp: P: 1751032536391962 usec, L: 0
0627 13:55:40.212542 (+ 192us) write_op.cc:270] Start()
0627 13:55:40.212567 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536394754 usec, L: 0
0627 13:55:40.212749 (+ 182us) write_op.cc:270] Start()
0627 13:55:40.212775 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536399249 usec, L: 0
0627 13:55:40.213328 (+ 553us) write_op.cc:270] Start()
0627 13:55:40.213358 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536415533 usec, L: 0
0627 13:55:40.213635 (+ 277us) write_op.cc:270] Start()
0627 13:55:40.213663 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536460606 usec, L: 0
0627 13:55:40.213895 (+ 232us) write_op.cc:270] Start()
0627 13:55:40.213921 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536474507 usec, L: 0
0627 13:55:40.214428 (+ 507us) write_op.cc:270] Start()
0627 13:55:40.214484 (+ 56us) write_op.cc:276] Timestamp: P: 1751032536507976 usec, L: 0
0627 13:55:40.215848 (+ 1364us) write_op.cc:270] Start()
0627 13:55:40.215880 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536509836 usec, L: 0
0627 13:55:40.216148 (+ 268us) write_op.cc:270] Start()
0627 13:55:40.216175 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536516690 usec, L: 0
0627 13:55:40.216397 (+ 222us) write_op.cc:270] Start()
0627 13:55:40.216422 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536529457 usec, L: 0
0627 13:55:40.220771 (+ 4349us) write_op.cc:270] Start()
0627 13:55:40.220807 (+ 36us) write_op.cc:276] Timestamp: P: 1751032536580096 usec, L: 0
0627 13:55:40.223221 (+ 2414us) write_op.cc:270] Start()
0627 13:55:40.223248 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536585672 usec, L: 0
0627 13:55:40.224953 (+ 1705us) write_op.cc:270] Start()
0627 13:55:40.224973 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536590308 usec, L: 0
0627 13:55:40.225152 (+ 179us) write_op.cc:270] Start()
0627 13:55:40.225177 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536617007 usec, L: 0
0627 13:55:40.230377 (+ 5200us) write_op.cc:270] Start()
0627 13:55:40.230403 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536635671 usec, L: 0
0627 13:55:40.240806 (+ 10403us) write_op.cc:270] Start()
0627 13:55:40.240825 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536639619 usec, L: 0
0627 13:55:40.245242 (+ 4417us) write_op.cc:270] Start()
0627 13:55:40.247174 (+ 1932us) write_op.cc:276] Timestamp: P: 1751032536657904 usec, L: 0
0627 13:55:40.249113 (+ 1939us) write_op.cc:270] Start()
0627 13:55:40.249144 (+ 31us) write_op.cc:276] Timestamp: P: 1751032536682454 usec, L: 0
0627 13:55:40.249370 (+ 226us) write_op.cc:270] Start()
0627 13:55:40.249408 (+ 38us) write_op.cc:276] Timestamp: P: 1751032536705695 usec, L: 0
0627 13:55:40.251016 (+ 1608us) write_op.cc:270] Start()
0627 13:55:40.251044 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536728197 usec, L: 0
0627 13:55:40.256219 (+ 5175us) write_op.cc:270] Start()
0627 13:55:40.256250 (+ 31us) write_op.cc:276] Timestamp: P: 1751032536739850 usec, L: 0
0627 13:55:40.259408 (+ 3158us) write_op.cc:270] Start()
0627 13:55:40.259430 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536760913 usec, L: 0
0627 13:55:40.264112 (+ 4682us) write_op.cc:270] Start()
0627 13:55:40.264139 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536784168 usec, L: 0
0627 13:55:40.306658 (+ 42519us) write_op.cc:270] Start()
0627 13:55:40.306682 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536829008 usec, L: 0
0627 13:55:40.334080 (+ 27398us) write_op.cc:270] Start()
0627 13:55:40.334111 (+ 31us) write_op.cc:276] Timestamp: P: 1751032536829642 usec, L: 0
0627 13:55:40.366950 (+ 32839us) write_op.cc:270] Start()
0627 13:55:40.366983 (+ 33us) write_op.cc:276] Timestamp: P: 1751032536849042 usec, L: 0
0627 13:55:40.368075 (+ 1092us) write_op.cc:270] Start()
0627 13:55:40.368107 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536852824 usec, L: 0
0627 13:55:40.370579 (+ 2472us) write_op.cc:270] Start()
0627 13:55:40.373119 (+ 2540us) write_op.cc:276] Timestamp: P: 1751032536900600 usec, L: 0
0627 13:55:40.375256 (+ 2137us) write_op.cc:270] Start()
0627 13:55:40.375283 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536908142 usec, L: 0
0627 13:55:40.382120 (+ 6837us) write_op.cc:270] Start()
0627 13:55:40.382145 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536910130 usec, L: 0
0627 13:55:40.382369 (+ 224us) write_op.cc:270] Start()
0627 13:55:40.382396 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536913233 usec, L: 0
0627 13:55:40.382576 (+ 180us) write_op.cc:270] Start()
0627 13:55:40.382593 (+ 17us) write_op.cc:276] Timestamp: P: 1751032536967233 usec, L: 0
0627 13:55:40.382795 (+ 202us) write_op.cc:270] Start()
0627 13:55:40.382822 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536986199 usec, L: 0
0627 13:55:40.382966 (+ 144us) write_op.cc:270] Start()
0627 13:55:40.382982 (+ 16us) write_op.cc:276] Timestamp: P: 1751032536986861 usec, L: 0
0627 13:55:40.383083 (+ 101us) write_op.cc:270] Start()
0627 13:55:40.383096 (+ 13us) write_op.cc:276] Timestamp: P: 1751032536995330 usec, L: 0
0627 13:55:40.383193 (+ 97us) write_op.cc:270] Start()
0627 13:55:40.383205 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537043461 usec, L: 0
0627 13:55:40.383294 (+ 89us) write_op.cc:270] Start()
0627 13:55:40.383306 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537052159 usec, L: 0
0627 13:55:40.383383 (+ 77us) write_op.cc:270] Start()
0627 13:55:40.383394 (+ 11us) write_op.cc:276] Timestamp: P: 1751032537060238 usec, L: 0
0627 13:55:40.383501 (+ 107us) write_op.cc:270] Start()
0627 13:55:40.383514 (+ 13us) write_op.cc:276] Timestamp: P: 1751032537062085 usec, L: 0
0627 13:55:40.383605 (+ 91us) write_op.cc:270] Start()
0627 13:55:40.383617 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537092963 usec, L: 0
0627 13:55:40.383722 (+ 105us) write_op.cc:270] Start()
0627 13:55:40.383734 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537169801 usec, L: 0
0627 13:55:40.390260 (+ 6526us) write_op.cc:270] Start()
0627 13:55:40.390294 (+ 34us) write_op.cc:276] Timestamp: P: 1751032537207021 usec, L: 0
0627 13:55:40.390520 (+ 226us) write_op.cc:270] Start()
0627 13:55:40.390546 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537232371 usec, L: 0
0627 13:55:40.390720 (+ 174us) write_op.cc:270] Start()
0627 13:55:40.390738 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537250988 usec, L: 0
0627 13:55:40.390898 (+ 160us) write_op.cc:270] Start()
0627 13:55:40.390919 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537254663 usec, L: 0
0627 13:55:40.391064 (+ 145us) write_op.cc:270] Start()
0627 13:55:40.391085 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537284747 usec, L: 0
0627 13:55:40.391274 (+ 189us) write_op.cc:270] Start()
0627 13:55:40.391297 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537297176 usec, L: 0
0627 13:55:40.391455 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.391503 (+ 48us) write_op.cc:276] Timestamp: P: 1751032537323118 usec, L: 0
0627 13:55:40.405709 (+ 14206us) write_op.cc:270] Start()
0627 13:55:40.405746 (+ 37us) write_op.cc:276] Timestamp: P: 1751032537330678 usec, L: 0
0627 13:55:40.416266 (+ 10520us) write_op.cc:270] Start()
0627 13:55:40.416292 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537364519 usec, L: 0
0627 13:55:40.417290 (+ 998us) write_op.cc:270] Start()
0627 13:55:40.417314 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537374938 usec, L: 0
0627 13:55:40.421175 (+ 3861us) write_op.cc:270] Start()
0627 13:55:40.421202 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537382354 usec, L: 0
0627 13:55:40.425706 (+ 4504us) write_op.cc:270] Start()
0627 13:55:40.425738 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537422852 usec, L: 0
0627 13:55:40.428453 (+ 2715us) write_op.cc:270] Start()
0627 13:55:40.428480 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537427286 usec, L: 0
0627 13:55:40.429231 (+ 751us) write_op.cc:270] Start()
0627 13:55:40.429258 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537460019 usec, L: 0
0627 13:55:40.436039 (+ 6781us) write_op.cc:270] Start()
0627 13:55:40.436068 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537471840 usec, L: 0
0627 13:55:40.444994 (+ 8926us) write_op.cc:270] Start()
0627 13:55:40.445022 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537478697 usec, L: 0
0627 13:55:40.462456 (+ 17434us) write_op.cc:270] Start()
0627 13:55:40.462483 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537489516 usec, L: 0
0627 13:55:40.468447 (+ 5964us) write_op.cc:270] Start()
0627 13:55:40.468481 (+ 34us) write_op.cc:276] Timestamp: P: 1751032537517983 usec, L: 0
0627 13:55:40.472972 (+ 4491us) write_op.cc:270] Start()
0627 13:55:40.473004 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537538689 usec, L: 0
0627 13:55:40.477620 (+ 4616us) write_op.cc:270] Start()
0627 13:55:40.477649 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537546849 usec, L: 0
0627 13:55:40.495443 (+ 17794us) write_op.cc:270] Start()
0627 13:55:40.495475 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537553774 usec, L: 0
0627 13:55:40.497783 (+ 2308us) write_op.cc:270] Start()
0627 13:55:40.497810 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537589930 usec, L: 0
0627 13:55:40.514390 (+ 16580us) write_op.cc:270] Start()
0627 13:55:40.514428 (+ 38us) write_op.cc:276] Timestamp: P: 1751032537611939 usec, L: 0
0627 13:55:40.525289 (+ 10861us) write_op.cc:270] Start()
0627 13:55:40.525314 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537612548 usec, L: 0
0627 13:55:40.528432 (+ 3118us) write_op.cc:270] Start()
0627 13:55:40.528462 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537625559 usec, L: 0
0627 13:55:40.536182 (+ 7720us) write_op.cc:270] Start()
0627 13:55:40.536213 (+ 31us) write_op.cc:276] Timestamp: P: 1751032537689963 usec, L: 0
0627 13:55:40.545012 (+ 8799us) write_op.cc:270] Start()
0627 13:55:40.545053 (+ 41us) write_op.cc:276] Timestamp: P: 1751032537697434 usec, L: 0
0627 13:55:40.548994 (+ 3941us) write_op.cc:270] Start()
0627 13:55:40.549029 (+ 35us) write_op.cc:276] Timestamp: P: 1751032537705958 usec, L: 0
0627 13:55:40.567293 (+ 18264us) write_op.cc:270] Start()
0627 13:55:40.567331 (+ 38us) write_op.cc:276] Timestamp: P: 1751032537720147 usec, L: 0
0627 13:55:40.572537 (+ 5206us) write_op.cc:270] Start()
0627 13:55:40.572563 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537747413 usec, L: 0
0627 13:55:40.578218 (+ 5655us) write_op.cc:270] Start()
0627 13:55:40.578248 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537775956 usec, L: 0
0627 13:55:40.581721 (+ 3473us) write_op.cc:270] Start()
0627 13:55:40.581747 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537778376 usec, L: 0
0627 13:55:40.593822 (+ 12075us) write_op.cc:270] Start()
0627 13:55:40.593850 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537786833 usec, L: 0
0627 13:55:40.607218 (+ 13368us) write_op.cc:270] Start()
0627 13:55:40.607249 (+ 31us) write_op.cc:276] Timestamp: P: 1751032537822107 usec, L: 0
0627 13:55:40.609405 (+ 2156us) write_op.cc:270] Start()
0627 13:55:40.609430 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537838533 usec, L: 0
0627 13:55:40.611521 (+ 2091us) write_op.cc:270] Start()
0627 13:55:40.611549 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537848326 usec, L: 0
0627 13:55:40.617077 (+ 5528us) write_op.cc:270] Start()
0627 13:55:40.617097 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537864565 usec, L: 0
0627 13:55:40.632026 (+ 14929us) write_op.cc:270] Start()
0627 13:55:40.632057 (+ 31us) write_op.cc:276] Timestamp: P: 1751032537874278 usec, L: 0
0627 13:55:40.638981 (+ 6924us) write_op.cc:270] Start()
0627 13:55:40.639011 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537907350 usec, L: 0
0627 13:55:40.650833 (+ 11822us) write_op.cc:270] Start()
0627 13:55:40.650860 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537917974 usec, L: 0
0627 13:55:40.663054 (+ 12194us) write_op.cc:270] Start()
0627 13:55:40.663083 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537927889 usec, L: 0
0627 13:55:40.665677 (+ 2594us) write_op.cc:270] Start()
0627 13:55:40.665706 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537929600 usec, L: 0
0627 13:55:40.670112 (+ 4406us) write_op.cc:270] Start()
0627 13:55:40.670140 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537973150 usec, L: 0
0627 13:55:40.670348 (+ 208us) write_op.cc:270] Start()
0627 13:55:40.670366 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537973799 usec, L: 0
0627 13:55:40.670524 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.670542 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537984696 usec, L: 0
0627 13:55:40.670689 (+ 147us) write_op.cc:270] Start()
0627 13:55:40.670707 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537989389 usec, L: 0
0627 13:55:40.670868 (+ 161us) write_op.cc:270] Start()
0627 13:55:40.670886 (+ 18us) write_op.cc:276] Timestamp: P: 1751032538057121 usec, L: 0
0627 13:55:40.671048 (+ 162us) write_op.cc:270] Start()
0627 13:55:40.671070 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538077308 usec, L: 0
0627 13:55:40.671263 (+ 193us) write_op.cc:270] Start()
0627 13:55:40.671285 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538080211 usec, L: 0
0627 13:55:40.671474 (+ 189us) write_op.cc:270] Start()
0627 13:55:40.671494 (+ 20us) write_op.cc:276] Timestamp: P: 1751032538082286 usec, L: 0
0627 13:55:40.671695 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.671717 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538098364 usec, L: 0
0627 13:55:40.672048 (+ 331us) write_op.cc:270] Start()
0627 13:55:40.672071 (+ 23us) write_op.cc:276] Timestamp: P: 1751032538144043 usec, L: 0
0627 13:55:40.672365 (+ 294us) write_op.cc:270] Start()
0627 13:55:40.672386 (+ 21us) write_op.cc:276] Timestamp: P: 175103
I20250627 13:55:41.901845 9919 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:41.905058 9919 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:41.914076 9919 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:41.914810 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:41.915679 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:41.916294 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:41.916816 9919 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:42.142819 9919 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:42.144140 9919 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:42.144847 9919 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:42.145731 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1050 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:42.149006 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1050 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:42.154858 9760 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:42.156559 9919 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:41.460094 9830 rpcz_store.cc:276] 0627 13:55:39.716892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:39.720105 (+ 3213us) service_pool.cc:225] Handling call
0627 13:55:39.870730 (+150625us) raft_consensus.cc:1469] Updating replica for 262 ops
0627 13:55:40.142095 (+271365us) raft_consensus.cc:1532] Early marking committed up to index 760
0627 13:55:40.145082 (+ 2987us) raft_consensus.cc:1537] Triggering prepare for 262 ops
0627 13:55:40.146748 (+ 1666us) write_op.cc:270] Start()
0627 13:55:40.146773 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534353713 usec, L: 0
0627 13:55:40.149508 (+ 2735us) write_op.cc:270] Start()
0627 13:55:40.149536 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534356592 usec, L: 0
0627 13:55:40.150992 (+ 1456us) write_op.cc:270] Start()
0627 13:55:40.151019 (+ 27us) write_op.cc:276] Timestamp: P: 1751032534357570 usec, L: 0
0627 13:55:40.151202 (+ 183us) write_op.cc:270] Start()
0627 13:55:40.151220 (+ 18us) write_op.cc:276] Timestamp: P: 1751032534380557 usec, L: 0
0627 13:55:40.153690 (+ 2470us) write_op.cc:270] Start()
0627 13:55:40.154261 (+ 571us) write_op.cc:276] Timestamp: P: 1751032534435571 usec, L: 0
0627 13:55:40.156691 (+ 2430us) write_op.cc:270] Start()
0627 13:55:40.156716 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534439181 usec, L: 0
0627 13:55:40.169776 (+ 13060us) write_op.cc:270] Start()
0627 13:55:40.169823 (+ 47us) write_op.cc:276] Timestamp: P: 1751032534445193 usec, L: 0
0627 13:55:40.171700 (+ 1877us) write_op.cc:270] Start()
0627 13:55:40.171726 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534446434 usec, L: 0
0627 13:55:40.175259 (+ 3533us) write_op.cc:270] Start()
0627 13:55:40.175288 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534480461 usec, L: 0
0627 13:55:40.175499 (+ 211us) write_op.cc:270] Start()
0627 13:55:40.175519 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534530102 usec, L: 0
0627 13:55:40.175661 (+ 142us) write_op.cc:270] Start()
0627 13:55:40.175680 (+ 19us) write_op.cc:276] Timestamp: P: 1751032534541449 usec, L: 0
0627 13:55:40.175797 (+ 117us) write_op.cc:270] Start()
0627 13:55:40.175814 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534542626 usec, L: 0
0627 13:55:40.175937 (+ 123us) write_op.cc:270] Start()
0627 13:55:40.175954 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534571081 usec, L: 0
0627 13:55:40.176060 (+ 106us) write_op.cc:270] Start()
0627 13:55:40.176075 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534600910 usec, L: 0
0627 13:55:40.176211 (+ 136us) write_op.cc:270] Start()
0627 13:55:40.176227 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534606640 usec, L: 0
0627 13:55:40.176353 (+ 126us) write_op.cc:270] Start()
0627 13:55:40.176370 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534639149 usec, L: 0
0627 13:55:40.176499 (+ 129us) write_op.cc:270] Start()
0627 13:55:40.176515 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534668196 usec, L: 0
0627 13:55:40.176639 (+ 124us) write_op.cc:270] Start()
0627 13:55:40.176656 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534693811 usec, L: 0
0627 13:55:40.176781 (+ 125us) write_op.cc:270] Start()
0627 13:55:40.176798 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534735663 usec, L: 0
0627 13:55:40.176920 (+ 122us) write_op.cc:270] Start()
0627 13:55:40.176936 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534736222 usec, L: 0
0627 13:55:40.177069 (+ 133us) write_op.cc:270] Start()
0627 13:55:40.177085 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534773946 usec, L: 0
0627 13:55:40.177195 (+ 110us) write_op.cc:270] Start()
0627 13:55:40.177211 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534807115 usec, L: 0
0627 13:55:40.177324 (+ 113us) write_op.cc:270] Start()
0627 13:55:40.177340 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534858128 usec, L: 0
0627 13:55:40.177454 (+ 114us) write_op.cc:270] Start()
0627 13:55:40.177469 (+ 15us) write_op.cc:276] Timestamp: P: 1751032534859048 usec, L: 0
0627 13:55:40.177612 (+ 143us) write_op.cc:270] Start()
0627 13:55:40.177628 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534860990 usec, L: 0
0627 13:55:40.177773 (+ 145us) write_op.cc:270] Start()
0627 13:55:40.177799 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534882132 usec, L: 0
0627 13:55:40.177927 (+ 128us) write_op.cc:270] Start()
0627 13:55:40.177943 (+ 16us) write_op.cc:276] Timestamp: P: 1751032534956902 usec, L: 0
0627 13:55:40.178101 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.178120 (+ 19us) write_op.cc:276] Timestamp: P: 1751032534989329 usec, L: 0
0627 13:55:40.178301 (+ 181us) write_op.cc:270] Start()
0627 13:55:40.178319 (+ 18us) write_op.cc:276] Timestamp: P: 1751032534998348 usec, L: 0
0627 13:55:40.178482 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.178499 (+ 17us) write_op.cc:276] Timestamp: P: 1751032534999303 usec, L: 0
0627 13:55:40.178666 (+ 167us) write_op.cc:270] Start()
0627 13:55:40.178684 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535059825 usec, L: 0
0627 13:55:40.178853 (+ 169us) write_op.cc:270] Start()
0627 13:55:40.178871 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535082675 usec, L: 0
0627 13:55:40.179068 (+ 197us) write_op.cc:270] Start()
0627 13:55:40.179087 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535086905 usec, L: 0
0627 13:55:40.179250 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.179268 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535108943 usec, L: 0
0627 13:55:40.179446 (+ 178us) write_op.cc:270] Start()
0627 13:55:40.179464 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535114887 usec, L: 0
0627 13:55:40.179623 (+ 159us) write_op.cc:270] Start()
0627 13:55:40.179640 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535173101 usec, L: 0
0627 13:55:40.179806 (+ 166us) write_op.cc:270] Start()
0627 13:55:40.179823 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535177180 usec, L: 0
0627 13:55:40.179970 (+ 147us) write_op.cc:270] Start()
0627 13:55:40.179988 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535187915 usec, L: 0
0627 13:55:40.180144 (+ 156us) write_op.cc:270] Start()
0627 13:55:40.180161 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535191870 usec, L: 0
0627 13:55:40.180311 (+ 150us) write_op.cc:270] Start()
0627 13:55:40.180327 (+ 16us) write_op.cc:276] Timestamp: P: 1751032535229509 usec, L: 0
0627 13:55:40.180490 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.180509 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535252660 usec, L: 0
0627 13:55:40.180667 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.180684 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535258666 usec, L: 0
0627 13:55:40.180868 (+ 184us) write_op.cc:270] Start()
0627 13:55:40.180885 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535259856 usec, L: 0
0627 13:55:40.181034 (+ 149us) write_op.cc:270] Start()
0627 13:55:40.181051 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535292309 usec, L: 0
0627 13:55:40.181214 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.181232 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535310343 usec, L: 0
0627 13:55:40.181386 (+ 154us) write_op.cc:270] Start()
0627 13:55:40.181403 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535327119 usec, L: 0
0627 13:55:40.181572 (+ 169us) write_op.cc:270] Start()
0627 13:55:40.181589 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535331227 usec, L: 0
0627 13:55:40.181753 (+ 164us) write_op.cc:270] Start()
0627 13:55:40.181771 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535363209 usec, L: 0
0627 13:55:40.181920 (+ 149us) write_op.cc:270] Start()
0627 13:55:40.181938 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535382972 usec, L: 0
0627 13:55:40.205268 (+ 23330us) write_op.cc:270] Start()
0627 13:55:40.205313 (+ 45us) write_op.cc:276] Timestamp: P: 1751032535389238 usec, L: 0
0627 13:55:40.225916 (+ 20603us) write_op.cc:270] Start()
0627 13:55:40.225944 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535404130 usec, L: 0
0627 13:55:40.226614 (+ 670us) write_op.cc:270] Start()
0627 13:55:40.226640 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535407256 usec, L: 0
0627 13:55:40.232205 (+ 5565us) write_op.cc:270] Start()
0627 13:55:40.232232 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535439652 usec, L: 0
0627 13:55:40.237510 (+ 5278us) write_op.cc:270] Start()
0627 13:55:40.237538 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535441186 usec, L: 0
0627 13:55:40.239664 (+ 2126us) write_op.cc:270] Start()
0627 13:55:40.239699 (+ 35us) write_op.cc:276] Timestamp: P: 1751032535483897 usec, L: 0
0627 13:55:40.239890 (+ 191us) write_op.cc:270] Start()
0627 13:55:40.239915 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535504882 usec, L: 0
0627 13:55:40.240075 (+ 160us) write_op.cc:270] Start()
0627 13:55:40.240095 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535509081 usec, L: 0
0627 13:55:40.240239 (+ 144us) write_op.cc:270] Start()
0627 13:55:40.240260 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535516981 usec, L: 0
0627 13:55:40.240436 (+ 176us) write_op.cc:270] Start()
0627 13:55:40.240458 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535533066 usec, L: 0
0627 13:55:40.240657 (+ 199us) write_op.cc:270] Start()
0627 13:55:40.240710 (+ 53us) write_op.cc:276] Timestamp: P: 1751032535604017 usec, L: 0
0627 13:55:40.253409 (+ 12699us) write_op.cc:270] Start()
0627 13:55:40.253434 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535610653 usec, L: 0
0627 13:55:40.256562 (+ 3128us) write_op.cc:270] Start()
0627 13:55:40.256585 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535611228 usec, L: 0
0627 13:55:40.256780 (+ 195us) write_op.cc:270] Start()
0627 13:55:40.257799 (+ 1019us) write_op.cc:276] Timestamp: P: 1751032535619538 usec, L: 0
0627 13:55:40.258054 (+ 255us) write_op.cc:270] Start()
0627 13:55:40.258078 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535670256 usec, L: 0
0627 13:55:40.264945 (+ 6867us) write_op.cc:270] Start()
0627 13:55:40.264968 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535673326 usec, L: 0
0627 13:55:40.265183 (+ 215us) write_op.cc:270] Start()
0627 13:55:40.265205 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535685496 usec, L: 0
0627 13:55:40.265422 (+ 217us) write_op.cc:270] Start()
0627 13:55:40.265444 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535717938 usec, L: 0
0627 13:55:40.265630 (+ 186us) write_op.cc:270] Start()
0627 13:55:40.265650 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535727822 usec, L: 0
0627 13:55:40.265853 (+ 203us) write_op.cc:270] Start()
0627 13:55:40.265877 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535734926 usec, L: 0
0627 13:55:40.266064 (+ 187us) write_op.cc:270] Start()
0627 13:55:40.266088 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535740177 usec, L: 0
0627 13:55:40.266308 (+ 220us) write_op.cc:270] Start()
0627 13:55:40.266329 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535778601 usec, L: 0
0627 13:55:40.266517 (+ 188us) write_op.cc:270] Start()
0627 13:55:40.266537 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535801296 usec, L: 0
0627 13:55:40.308407 (+ 41870us) write_op.cc:270] Start()
0627 13:55:40.321173 (+ 12766us) write_op.cc:276] Timestamp: P: 1751032535804772 usec, L: 0
0627 13:55:40.332290 (+ 11117us) write_op.cc:270] Start()
0627 13:55:40.332321 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535808464 usec, L: 0
0627 13:55:40.332534 (+ 213us) write_op.cc:270] Start()
0627 13:55:40.332561 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535847675 usec, L: 0
0627 13:55:40.332725 (+ 164us) write_op.cc:270] Start()
0627 13:55:40.332746 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535858462 usec, L: 0
0627 13:55:40.363675 (+ 30929us) write_op.cc:270] Start()
0627 13:55:40.363698 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535863737 usec, L: 0
0627 13:55:40.365793 (+ 2095us) write_op.cc:270] Start()
0627 13:55:40.365825 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535871699 usec, L: 0
0627 13:55:40.370093 (+ 4268us) write_op.cc:270] Start()
0627 13:55:40.373972 (+ 3879us) write_op.cc:276] Timestamp: P: 1751032535902802 usec, L: 0
0627 13:55:40.376446 (+ 2474us) write_op.cc:270] Start()
0627 13:55:40.376469 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535924040 usec, L: 0
0627 13:55:40.376632 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.376665 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535938482 usec, L: 0
0627 13:55:40.384217 (+ 7552us) write_op.cc:270] Start()
0627 13:55:40.384237 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535972717 usec, L: 0
0627 13:55:40.395253 (+ 11016us) write_op.cc:270] Start()
0627 13:55:40.395282 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535993017 usec, L: 0
0627 13:55:40.395466 (+ 184us) write_op.cc:270] Start()
0627 13:55:40.395491 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536032004 usec, L: 0
0627 13:55:40.398297 (+ 2806us) write_op.cc:270] Start()
0627 13:55:40.398326 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536035958 usec, L: 0
0627 13:55:40.398537 (+ 211us) write_op.cc:270] Start()
0627 13:55:40.398562 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536053169 usec, L: 0
0627 13:55:40.398784 (+ 222us) write_op.cc:270] Start()
0627 13:55:40.398807 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536112889 usec, L: 0
0627 13:55:40.398961 (+ 154us) write_op.cc:270] Start()
0627 13:55:40.398981 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536131679 usec, L: 0
0627 13:55:40.400467 (+ 1486us) write_op.cc:270] Start()
0627 13:55:40.400495 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536137769 usec, L: 0
0627 13:55:40.400696 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.400718 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536150509 usec, L: 0
0627 13:55:40.400915 (+ 197us) write_op.cc:270] Start()
0627 13:55:40.400937 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536216449 usec, L: 0
0627 13:55:40.401093 (+ 156us) write_op.cc:270] Start()
0627 13:55:40.401116 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536250810 usec, L: 0
0627 13:55:40.401292 (+ 176us) write_op.cc:270] Start()
0627 13:55:40.401315 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536256978 usec, L: 0
0627 13:55:40.401457 (+ 142us) write_op.cc:270] Start()
0627 13:55:40.401476 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536263996 usec, L: 0
0627 13:55:40.401656 (+ 180us) write_op.cc:270] Start()
0627 13:55:40.401697 (+ 41us) write_op.cc:276] Timestamp: P: 1751032536267756 usec, L: 0
0627 13:55:40.413563 (+ 11866us) write_op.cc:270] Start()
0627 13:55:40.413598 (+ 35us) write_op.cc:276] Timestamp: P: 1751032536330378 usec, L: 0
0627 13:55:40.413825 (+ 227us) write_op.cc:270] Start()
0627 13:55:40.413851 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536331531 usec, L: 0
0627 13:55:40.416060 (+ 2209us) write_op.cc:270] Start()
0627 13:55:40.416082 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536332217 usec, L: 0
0627 13:55:40.416639 (+ 557us) write_op.cc:270] Start()
0627 13:55:40.416663 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536348395 usec, L: 0
0627 13:55:40.416825 (+ 162us) write_op.cc:270] Start()
0627 13:55:40.416846 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536404032 usec, L: 0
0627 13:55:40.417040 (+ 194us) write_op.cc:270] Start()
0627 13:55:40.417063 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536405152 usec, L: 0
0627 13:55:40.429340 (+ 12277us) write_op.cc:270] Start()
0627 13:55:40.429360 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536405835 usec, L: 0
0627 13:55:40.429647 (+ 287us) write_op.cc:270] Start()
0627 13:55:40.429669 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536415756 usec, L: 0
0627 13:55:40.431828 (+ 2159us) write_op.cc:270] Start()
0627 13:55:40.431857 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536459599 usec, L: 0
0627 13:55:40.449574 (+ 17717us) write_op.cc:270] Start()
0627 13:55:40.449606 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536460289 usec, L: 0
0627 13:55:40.466886 (+ 17280us) write_op.cc:270] Start()
0627 13:55:40.466913 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536488969 usec, L: 0
0627 13:55:40.472115 (+ 5202us) write_op.cc:270] Start()
0627 13:55:40.472141 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536510460 usec, L: 0
0627 13:55:40.472350 (+ 209us) write_op.cc:270] Start()
0627 13:55:40.472371 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536519481 usec, L: 0
0627 13:55:40.477674 (+ 5303us) write_op.cc:270] Start()
0627 13:55:40.477704 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536531641 usec, L: 0
0627 13:55:40.480694 (+ 2990us) write_op.cc:270] Start()
0627 13:55:40.480721 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536577245 usec, L: 0
0627 13:55:40.480950 (+ 229us) write_op.cc:270] Start()
0627 13:55:40.480989 (+ 39us) write_op.cc:276] Timestamp: P: 1751032536582513 usec, L: 0
0627 13:55:40.487693 (+ 6704us) write_op.cc:270] Start()
0627 13:55:40.487726 (+ 33us) write_op.cc:276] Timestamp: P: 1751032536586722 usec, L: 0
0627 13:55:40.487944 (+ 218us) write_op.cc:270] Start()
0627 13:55:40.487966 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536590110 usec, L: 0
0627 13:55:40.488168 (+ 202us) write_op.cc:270] Start()
0627 13:55:40.488188 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536636565 usec, L: 0
0627 13:55:40.488359 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.488380 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536641353 usec, L: 0
0627 13:55:40.488551 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.488572 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536655791 usec, L: 0
0627 13:55:40.488788 (+ 216us) write_op.cc:270] Start()
0627 13:55:40.488811 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536685686 usec, L: 0
0627 13:55:40.488991 (+ 180us) write_op.cc:270] Start()
0627 13:55:40.489011 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536709031 usec, L: 0
0627 13:55:40.489186 (+ 175us) write_op.cc:270] Start()
0627 13:55:40.489207 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536725185 usec, L: 0
0627 13:55:40.489374 (+ 167us) write_op.cc:270] Start()
0627 13:55:40.489394 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536743867 usec, L: 0
0627 13:55:40.489565 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.489587 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536766265 usec, L: 0
0627 13:55:40.489786 (+ 199us) write_op.cc:270] Start()
0627 13:55:40.489807 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536786179 usec, L: 0
0627 13:55:40.490002 (+ 195us) write_op.cc:270] Start()
0627 13:55:40.490023 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536808769 usec, L: 0
0627 13:55:40.494139 (+ 4116us) write_op.cc:270] Start()
0627 13:55:40.494164 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536855384 usec, L: 0
0627 13:55:40.496068 (+ 1904us) write_op.cc:270] Start()
0627 13:55:40.496089 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536866901 usec, L: 0
0627 13:55:40.496271 (+ 182us) write_op.cc:270] Start()
0627 13:55:40.496292 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536867962 usec, L: 0
0627 13:55:40.496474 (+ 182us) write_op.cc:270] Start()
0627 13:55:40.496520 (+ 46us) write_op.cc:276] Timestamp: P: 1751032536874939 usec, L: 0
0627 13:55:40.524316 (+ 27796us) write_op.cc:270] Start()
0627 13:55:40.524341 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536906569 usec, L: 0
0627 13:55:40.536670 (+ 12329us) write_op.cc:270] Start()
0627 13:55:40.536702 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536911085 usec, L: 0
0627 13:55:40.556218 (+ 19516us) write_op.cc:270] Start()
0627 13:55:40.556257 (+ 39us) write_op.cc:276] Timestamp: P: 1751032536917606 usec, L: 0
0627 13:55:40.568741 (+ 12484us) write_op.cc:270] Start()
0627 13:55:40.569298 (+ 557us) write_op.cc:276] Timestamp: P: 1751032536969803 usec, L: 0
0627 13:55:40.569537 (+ 239us) write_op.cc:270] Start()
0627 13:55:40.569566 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537000870 usec, L: 0
0627 13:55:40.569770 (+ 204us) write_op.cc:270] Start()
0627 13:55:40.569792 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537004315 usec, L: 0
0627 13:55:40.569972 (+ 180us) write_op.cc:270] Start()
0627 13:55:40.569994 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537008436 usec, L: 0
0627 13:55:40.573173 (+ 3179us) write_op.cc:270] Start()
0627 13:55:40.573202 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537043442 usec, L: 0
0627 13:55:40.573396 (+ 194us) write_op.cc:270] Start()
0627 13:55:40.573419 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537066757 usec, L: 0
0627 13:55:40.575431 (+ 2012us) write_op.cc:270] Start()
0627 13:55:40.575461 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537076355 usec, L: 0
0627 13:55:40.575665 (+ 204us) write_op.cc:270] Start()
0627 13:55:40.575686 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537095459 usec, L: 0
0627 13:55:40.575824 (+ 138us) write_op.cc:270] Start()
0627 13:55:40.575841 (+ 17us) write_op.cc:276] Timestamp: P: 1751032537096134 usec, L: 0
0627 13:55:40.575992 (+ 151us) write_op.cc:270] Start()
0627 13:55:40.576010 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537171240 usec, L: 0
0627 13:55:40.576158 (+ 148us) write_op.cc:270] Start()
0627 13:55:40.576175 (+ 17us) write_op.cc:276] Timestamp: P: 1751032537202764 usec, L: 0
0627 13:55:40.576311 (+ 136us) write_op.cc:270] Start()
0627 13:55:40.576332 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537232897 usec, L: 0
0627 13:55:40.576464 (+ 132us) write_op.cc:270] Start()
0627 13:55:40.599306 (+ 22842us) write_op.cc:276] Timestamp: P: 1751032537233788 usec, L: 0
0627 13:55:40.601475 (+ 2169us) write_op.cc:270] Start()
0627 13:55:40.601501 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537260111 usec, L: 0
0627 13:55:40.601818 (+ 317us) write_op.cc:270] Start()
0627 13:55:40.601846 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537282534 usec, L: 0
0627 13:55:40.639428 (+ 37582us) write_op.cc:270] Start()
0627 13:55:40.639448 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537296978 usec, L: 0
0627 13:55:40.647345 (+ 7897us) write_op.cc:270] Start()
0627 13:55:40.647373 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537321219 usec, L: 0
0627 13:55:40.649328 (+ 1955us) write_op.cc:270] Start()
0627 13:55:40.649354 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537332304 usec, L: 0
0627 13:55:40.664195 (+ 14841us) write_op.cc:270] Start()
0627 13:55:40.664221 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537365127 usec, L: 0
0627 13:55:40.682088 (+ 17867us) write_op.cc:270] Start()
0627 13:55:40.682114 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537375114 usec, L: 0
0627 13:55:40.682329 (+ 215us) write_op.cc:270] Start()
0627 13:55:40.682351 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537389753 usec, L: 0
0627 13:55:40.686725 (+ 4374us) write_op.cc:270] Start()
0627 13:55:40.686758 (+ 33us) write_op.cc:276] Timestamp: P: 1751032537421523 usec, L: 0
0627 13:55:40.688823 (+ 2065us) write_op.cc:270] Start()
0627 13:55:40.688850 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537426435 usec, L: 0
0627 13:55:40.689066 (+ 216us) write_op.cc:270] Start()
0627 13:55:40.689089 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537463096 usec, L: 0
0627 13:55:40.689300 (+ 211us) write_op.cc:270] Start()
0627 13:55:40.689323 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537469740 usec, L: 0
0627 13:55:40.689521 (+ 198us) write_op.cc:270] Start()
0627 13:55:40.689545 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537474394 usec, L: 0
0627 13:55:40.689742 (+ 197us) write_op.cc:270] Start()
0627 13:55:40.689764 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537494410 usec, L: 0
0627 13:55:40.689960 (+ 196us) write_op.cc:270] Start()
0627 13:55:40.689984 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537535682 usec, L: 0
0627 13:55:40.690210 (+ 226us) write_op.cc:270] Start()
0627 13:55:40.690246 (+ 36us) write_op.cc:276] Timestamp: P: 1751032537544836 usec, L: 0
0627 13:55:40.690429 (+ 183us) write_op.cc:270] Start()
0627 13:55:40.690452 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537547379 usec, L: 0
0627 13:55:40.690638 (+ 186us) write_op.cc:270] Start()
0627 13:55:40.690660 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537555721 usec, L: 0
0627 13:55:40.690847 (+ 187us) write_op.cc:270] Start()
0627 13:55:40.690870 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537591703 usec, L: 0
0627 13:55:40.691080 (+ 210us) write_op.cc:270] Start()
0627 13:55:40.691102 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537608640 usec, L: 0
0627 13:55:40.691275 (+ 173us) write_op.cc:270] Start()
0627 13:55:40.691297 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537621861 usec, L: 0
0627 13:55:40.691446 (+ 149us) write_op.cc:270] Start()
0627 13:55:40.691468 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537626921 usec, L: 0
0627 13:55:40.691628 (+ 160us) write_op.cc:270] Start()
0627 13:55:40.691650 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537692020 usec, L: 0
0627 13:55:40.691836 (+ 186us) write_op.cc:270] Start()
0627 13:55:40.691857 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537699709 usec, L: 0
0627 13:55:40.691999 (+ 142us) write_op.cc:270] Start()
0627 13:55:40.692017 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537708409 usec, L: 0
0627 13:55:40.704050 (+ 12033us) write_op.cc:270] Start()
0627 13:55:40.704077 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537720380 usec, L: 0
0627 13:55:40.704278 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.704298 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537757581 usec, L: 0
0627 13:55:40.709035 (+ 4737us) write_op.cc:270] Start()
0627 13:55:40.709051 (+ 16us) write_op.cc:276] Timestamp: P: 1751032537770016 usec, L: 0
0627 13:55:40.712065 (+ 3014us) write_op.cc:270] Start()
0627 13:55:40.712093 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537777757 usec, L: 0
0627 13:55:40.738352 (+ 26259us) write_op.cc:270] Start()
0627 13:55:40.738388 (+ 36us) write_op.cc:276] Timestamp: P: 1751032537782982 usec, L: 0
0627 13:55:40.749545 (+ 11157us) write_op.cc:270] Start()
0627 13:55:40.749586 (+ 41us) write_op.cc:276] Timestamp: P: 1751032537829232 usec, L: 0
0627 13:55:40.753408 (+ 3822us) write_op.cc:270] Start()
0627 13:55:40.753440 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537844447 usec, L: 0
0627 13:55:40.759091 (+ 5651us) write_op.cc:270] Start()
0627 13:55:40.759114 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537849669 usec, L: 0
0627 13:55:40.759315 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.759397 (+ 82us) write_op.cc:276] Timestamp: P: 1751032537869235 usec, L: 0
0627 13:55:40.759621 (+ 224us) write_op.cc:270] Start()
0627 13:55:40.759640 (+ 19us) write_op.cc:276] Timestamp: P: 1751032537875176 usec, L: 0
0627 13:55:40.760657 (+ 1017us) write_op.cc:270] Start()
0627 13:55:40.760682 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537913854 usec, L: 0
0627 13:55:40.764058 (+ 3376us) write_op.cc:270] Start()
0627 13:55:40.764081 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537914854 usec, L: 0
0627 13:55:40.802718 (+ 38637us) write_op.cc:270] Start()
0627 13:55:40.802746 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537921160 usec, L: 0
0627 13:55:40.803335 (+ 589us) write_op.cc:270] Start()
0627 13:55:40.803359 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537930440 usec, L: 0
0627 13:55:40.803530 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.803566 (+ 36us) write_op.cc:276] Timestamp: P: 1751032537975367 usec, L: 0
0627 13:55:40.803747 (+ 181us) write_op.cc:270] Start()
0627 13:55:40.803768 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537976589 usec, L: 0
0627 13:55:40.803926 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.803949 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537981370 usec, L: 0
0627 13:55:40.804105 (+ 156us) write_op.cc:270] Start()
0627 13:55:40.804126 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537991459 usec, L: 0
0627 13:55:40.804268 (+ 142us) write_op.cc:270] Start()
0627 13:55:40.804282 (+ 14us) write_op.cc:276] Timestamp: P: 1751032538048160 usec, L: 0
0627 13:55:40.804385 (+ 103us) write_op.cc:270] Start()
0627 13:55:40.804404 (+ 19us) write_op.cc:276] Timestamp: P: 1751032538068250 usec, L: 0
0627 13:55:40.806998 (+ 2594us) write_op.cc:270] Start()
0627 13:55:40.807026 (+ 28us) write_op.cc:276] Timestamp: P: 1751032538082691 usec, L: 0
0627 13:55:40.807248 (+ 222us) write_op.cc:270] Start()
0627 13:55:40.807272 (+ 24us) write_op.cc:276] Timestamp: P: 1751032538084169 usec, L: 0
0627 13:55:40.833883 (+ 26611us) write_op.cc:270] Start()
0627 13:55:40.833911 (+ 28us) write_op.cc:276] Timestamp: P: 1751032538100722 usec, L: 0
0627 13:55:40.837124 (+ 3213us) write_op.cc:270] Start()
0627 13:55:40.837152 (+ 28us) write_op.cc:276] Timestamp: P: 1751032538150280 usec, L: 0
0627 13:55:40.859484 (+ 22332us) write_op.cc:270] Start()
0627 13:55:40.859516 (+ 32us) write_op.cc:276] Timestamp: P: 1751032538160275 usec, L: 0
0627 13:55:40.859752 (+ 236us) write_op.cc:270] Start()
0627 13:55:40.859783 (+ 31us) write_op.cc:276] Timestamp: P: 1751032538165654 usec, L: 0
0627 13:55:40.865344 (+ 5561us) write_op.cc:270] Start()
0627 13:55:40.865363 (+ 19us) write_op.cc:276] Timestamp: P: 1751032538173506 usec, L: 0
0627 13:55:40.866577 (+ 1214us) write_op.cc:270] Start()
0627 13:55:40.866601 (+ 24us) write_op.cc:276] Timestamp: P: 1751032538230249 usec, L: 0
0627 13:55:40.873145 (+ 6544us) write_op.cc:270] Start()
0627 13:55:40.873166 (+ 21us) write_op.cc:276] Timestamp: P: 175103
I20250627 13:55:41.734587 9826 rpcz_store.cc:276] 0627 13:55:39.985953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:39.986014 (+ 61us) service_pool.cc:225] Handling call
0627 13:55:40.117789 (+131775us) raft_consensus.cc:1469] Updating replica for 278 ops
0627 13:55:40.258009 (+140220us) raft_consensus.cc:1532] Early marking committed up to index 772
0627 13:55:40.258889 (+ 880us) raft_consensus.cc:1537] Triggering prepare for 278 ops
0627 13:55:40.258999 (+ 110us) write_op.cc:270] Start()
0627 13:55:40.259022 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534571210 usec, L: 0
0627 13:55:40.259240 (+ 218us) write_op.cc:270] Start()
0627 13:55:40.259269 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534598704 usec, L: 0
0627 13:55:40.305534 (+ 46265us) write_op.cc:270] Start()
0627 13:55:40.305568 (+ 34us) write_op.cc:276] Timestamp: P: 1751032534610342 usec, L: 0
0627 13:55:40.305792 (+ 224us) write_op.cc:270] Start()
0627 13:55:40.305812 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534628378 usec, L: 0
0627 13:55:40.320892 (+ 15080us) write_op.cc:270] Start()
0627 13:55:40.320923 (+ 31us) write_op.cc:276] Timestamp: P: 1751032534666380 usec, L: 0
0627 13:55:40.322821 (+ 1898us) write_op.cc:270] Start()
0627 13:55:40.322851 (+ 30us) write_op.cc:276] Timestamp: P: 1751032534709460 usec, L: 0
0627 13:55:40.336543 (+ 13692us) write_op.cc:270] Start()
0627 13:55:40.346378 (+ 9835us) write_op.cc:276] Timestamp: P: 1751032534719224 usec, L: 0
0627 13:55:40.355591 (+ 9213us) write_op.cc:270] Start()
0627 13:55:40.355620 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534721352 usec, L: 0
0627 13:55:40.358507 (+ 2887us) write_op.cc:270] Start()
0627 13:55:40.358541 (+ 34us) write_op.cc:276] Timestamp: P: 1751032534758724 usec, L: 0
0627 13:55:40.363377 (+ 4836us) write_op.cc:270] Start()
0627 13:55:40.363404 (+ 27us) write_op.cc:276] Timestamp: P: 1751032534801791 usec, L: 0
0627 13:55:40.367332 (+ 3928us) write_op.cc:270] Start()
0627 13:55:40.367367 (+ 35us) write_op.cc:276] Timestamp: P: 1751032534837133 usec, L: 0
0627 13:55:40.370323 (+ 2956us) write_op.cc:270] Start()
0627 13:55:40.370347 (+ 24us) write_op.cc:276] Timestamp: P: 1751032534840482 usec, L: 0
0627 13:55:40.376219 (+ 5872us) write_op.cc:270] Start()
0627 13:55:40.376245 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534862821 usec, L: 0
0627 13:55:40.377548 (+ 1303us) write_op.cc:270] Start()
0627 13:55:40.377573 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534875954 usec, L: 0
0627 13:55:40.377754 (+ 181us) write_op.cc:270] Start()
0627 13:55:40.377774 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534959576 usec, L: 0
0627 13:55:40.392561 (+ 14787us) write_op.cc:270] Start()
0627 13:55:40.400391 (+ 7830us) write_op.cc:276] Timestamp: P: 1751032534989821 usec, L: 0
0627 13:55:40.400612 (+ 221us) write_op.cc:270] Start()
0627 13:55:40.400634 (+ 22us) write_op.cc:276] Timestamp: P: 1751032534991033 usec, L: 0
0627 13:55:40.400823 (+ 189us) write_op.cc:270] Start()
0627 13:55:40.400844 (+ 21us) write_op.cc:276] Timestamp: P: 1751032534994188 usec, L: 0
0627 13:55:40.401056 (+ 212us) write_op.cc:270] Start()
0627 13:55:40.401082 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535060280 usec, L: 0
0627 13:55:40.401257 (+ 175us) write_op.cc:270] Start()
0627 13:55:40.401276 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535085415 usec, L: 0
0627 13:55:40.401420 (+ 144us) write_op.cc:270] Start()
0627 13:55:40.401451 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535117257 usec, L: 0
0627 13:55:40.401614 (+ 163us) write_op.cc:270] Start()
0627 13:55:40.401636 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535118521 usec, L: 0
0627 13:55:40.401807 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.401829 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535119959 usec, L: 0
0627 13:55:40.408323 (+ 6494us) write_op.cc:270] Start()
0627 13:55:40.408354 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535177696 usec, L: 0
0627 13:55:40.421495 (+ 13141us) write_op.cc:270] Start()
0627 13:55:40.421525 (+ 30us) write_op.cc:276] Timestamp: P: 1751032535189028 usec, L: 0
0627 13:55:40.421727 (+ 202us) write_op.cc:270] Start()
0627 13:55:40.421749 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535192832 usec, L: 0
0627 13:55:40.421904 (+ 155us) write_op.cc:270] Start()
0627 13:55:40.421926 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535197533 usec, L: 0
0627 13:55:40.435536 (+ 13610us) write_op.cc:270] Start()
0627 13:55:40.435569 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535221057 usec, L: 0
0627 13:55:40.456344 (+ 20775us) write_op.cc:270] Start()
0627 13:55:40.456376 (+ 32us) write_op.cc:276] Timestamp: P: 1751032535251707 usec, L: 0
0627 13:55:40.456624 (+ 248us) write_op.cc:270] Start()
0627 13:55:40.456651 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535254042 usec, L: 0
0627 13:55:40.456809 (+ 158us) write_op.cc:270] Start()
0627 13:55:40.456830 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535261466 usec, L: 0
0627 13:55:40.457018 (+ 188us) write_op.cc:270] Start()
0627 13:55:40.457040 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535290061 usec, L: 0
0627 13:55:40.457235 (+ 195us) write_op.cc:270] Start()
0627 13:55:40.457261 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535310189 usec, L: 0
0627 13:55:40.457454 (+ 193us) write_op.cc:270] Start()
0627 13:55:40.457476 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535321023 usec, L: 0
0627 13:55:40.457659 (+ 183us) write_op.cc:270] Start()
0627 13:55:40.457682 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535333153 usec, L: 0
0627 13:55:40.457830 (+ 148us) write_op.cc:270] Start()
0627 13:55:40.457851 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535360875 usec, L: 0
0627 13:55:40.458004 (+ 153us) write_op.cc:270] Start()
0627 13:55:40.458026 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535379404 usec, L: 0
0627 13:55:40.458227 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.458252 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535386655 usec, L: 0
0627 13:55:40.458465 (+ 213us) write_op.cc:270] Start()
0627 13:55:40.458486 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535402367 usec, L: 0
0627 13:55:40.458687 (+ 201us) write_op.cc:270] Start()
0627 13:55:40.458717 (+ 30us) write_op.cc:276] Timestamp: P: 1751032535410629 usec, L: 0
0627 13:55:40.458910 (+ 193us) write_op.cc:270] Start()
0627 13:55:40.458931 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535426188 usec, L: 0
0627 13:55:40.459127 (+ 196us) write_op.cc:270] Start()
0627 13:55:40.459148 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535451240 usec, L: 0
0627 13:55:40.460071 (+ 923us) write_op.cc:270] Start()
0627 13:55:40.460100 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535485176 usec, L: 0
0627 13:55:40.468948 (+ 8848us) write_op.cc:270] Start()
0627 13:55:40.468973 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535499468 usec, L: 0
0627 13:55:40.469182 (+ 209us) write_op.cc:270] Start()
0627 13:55:40.469205 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535503028 usec, L: 0
0627 13:55:40.477895 (+ 8690us) write_op.cc:270] Start()
0627 13:55:40.477928 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535505460 usec, L: 0
0627 13:55:40.478189 (+ 261us) write_op.cc:270] Start()
0627 13:55:40.478218 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535521036 usec, L: 0
0627 13:55:40.498702 (+ 20484us) write_op.cc:270] Start()
0627 13:55:40.498729 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535582943 usec, L: 0
0627 13:55:40.524097 (+ 25368us) write_op.cc:270] Start()
0627 13:55:40.524134 (+ 37us) write_op.cc:276] Timestamp: P: 1751032535596932 usec, L: 0
0627 13:55:40.527713 (+ 3579us) write_op.cc:270] Start()
0627 13:55:40.527739 (+ 26us) write_op.cc:276] Timestamp: P: 1751032535601611 usec, L: 0
0627 13:55:40.534557 (+ 6818us) write_op.cc:270] Start()
0627 13:55:40.534594 (+ 37us) write_op.cc:276] Timestamp: P: 1751032535620089 usec, L: 0
0627 13:55:40.545334 (+ 10740us) write_op.cc:270] Start()
0627 13:55:40.545367 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535665143 usec, L: 0
0627 13:55:40.545638 (+ 271us) write_op.cc:270] Start()
0627 13:55:40.545660 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535667943 usec, L: 0
0627 13:55:40.545839 (+ 179us) write_op.cc:270] Start()
0627 13:55:40.545859 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535677895 usec, L: 0
0627 13:55:40.545997 (+ 138us) write_op.cc:270] Start()
0627 13:55:40.546017 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535715377 usec, L: 0
0627 13:55:40.546214 (+ 197us) write_op.cc:270] Start()
0627 13:55:40.546234 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535726086 usec, L: 0
0627 13:55:40.546399 (+ 165us) write_op.cc:270] Start()
0627 13:55:40.546416 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535731523 usec, L: 0
0627 13:55:40.546566 (+ 150us) write_op.cc:270] Start()
0627 13:55:40.546586 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535739107 usec, L: 0
0627 13:55:40.546726 (+ 140us) write_op.cc:270] Start()
0627 13:55:40.546746 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535781079 usec, L: 0
0627 13:55:40.546910 (+ 164us) write_op.cc:270] Start()
0627 13:55:40.546972 (+ 62us) write_op.cc:276] Timestamp: P: 1751032535791486 usec, L: 0
0627 13:55:40.547122 (+ 150us) write_op.cc:270] Start()
0627 13:55:40.547144 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535792507 usec, L: 0
0627 13:55:40.547295 (+ 151us) write_op.cc:270] Start()
0627 13:55:40.547316 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535797707 usec, L: 0
0627 13:55:40.547470 (+ 154us) write_op.cc:270] Start()
0627 13:55:40.547491 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535850205 usec, L: 0
0627 13:55:40.567596 (+ 20105us) write_op.cc:270] Start()
0627 13:55:40.567619 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535850868 usec, L: 0
0627 13:55:40.577599 (+ 9980us) write_op.cc:270] Start()
0627 13:55:40.577629 (+ 30us) write_op.cc:276] Timestamp: P: 1751032535851464 usec, L: 0
0627 13:55:40.580378 (+ 2749us) write_op.cc:270] Start()
0627 13:55:40.580403 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535878793 usec, L: 0
0627 13:55:40.593365 (+ 12962us) write_op.cc:270] Start()
0627 13:55:40.593400 (+ 35us) write_op.cc:276] Timestamp: P: 1751032535901376 usec, L: 0
0627 13:55:40.594561 (+ 1161us) write_op.cc:270] Start()
0627 13:55:40.594589 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535919963 usec, L: 0
0627 13:55:40.598784 (+ 4195us) write_op.cc:270] Start()
0627 13:55:40.598813 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535935724 usec, L: 0
0627 13:55:40.601136 (+ 2323us) write_op.cc:270] Start()
0627 13:55:40.601163 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535959520 usec, L: 0
0627 13:55:40.616631 (+ 15468us) write_op.cc:270] Start()
0627 13:55:40.616665 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535988464 usec, L: 0
0627 13:55:40.616877 (+ 212us) write_op.cc:270] Start()
0627 13:55:40.616900 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536038680 usec, L: 0
0627 13:55:40.632257 (+ 15357us) write_op.cc:270] Start()
0627 13:55:40.632278 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536055436 usec, L: 0
0627 13:55:40.632427 (+ 149us) write_op.cc:270] Start()
0627 13:55:40.632445 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536070062 usec, L: 0
0627 13:55:40.639194 (+ 6749us) write_op.cc:270] Start()
0627 13:55:40.639213 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536101677 usec, L: 0
0627 13:55:40.643309 (+ 4096us) write_op.cc:270] Start()
0627 13:55:40.643339 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536124006 usec, L: 0
0627 13:55:40.643610 (+ 271us) write_op.cc:270] Start()
0627 13:55:40.643636 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536132959 usec, L: 0
0627 13:55:40.643871 (+ 235us) write_op.cc:270] Start()
0627 13:55:40.643894 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536142176 usec, L: 0
0627 13:55:40.644090 (+ 196us) write_op.cc:270] Start()
0627 13:55:40.644112 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536159648 usec, L: 0
0627 13:55:40.644330 (+ 218us) write_op.cc:270] Start()
0627 13:55:40.644354 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536235966 usec, L: 0
0627 13:55:40.644565 (+ 211us) write_op.cc:270] Start()
0627 13:55:40.644588 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536249099 usec, L: 0
0627 13:55:40.644805 (+ 217us) write_op.cc:270] Start()
0627 13:55:40.644829 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536264094 usec, L: 0
0627 13:55:40.645027 (+ 198us) write_op.cc:270] Start()
0627 13:55:40.645049 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536264707 usec, L: 0
0627 13:55:40.645259 (+ 210us) write_op.cc:270] Start()
0627 13:55:40.645282 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536326097 usec, L: 0
0627 13:55:40.645735 (+ 453us) write_op.cc:270] Start()
0627 13:55:40.645759 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536327329 usec, L: 0
0627 13:55:40.645985 (+ 226us) write_op.cc:270] Start()
0627 13:55:40.646009 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536334632 usec, L: 0
0627 13:55:40.646224 (+ 215us) write_op.cc:270] Start()
0627 13:55:40.646247 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536335778 usec, L: 0
0627 13:55:40.648834 (+ 2587us) write_op.cc:270] Start()
0627 13:55:40.648863 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536393064 usec, L: 0
0627 13:55:40.666152 (+ 17289us) write_op.cc:270] Start()
0627 13:55:40.666177 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536399265 usec, L: 0
0627 13:55:40.666447 (+ 270us) write_op.cc:270] Start()
0627 13:55:40.666468 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536401409 usec, L: 0
0627 13:55:40.668195 (+ 1727us) write_op.cc:270] Start()
0627 13:55:40.668221 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536414578 usec, L: 0
0627 13:55:40.687255 (+ 19034us) write_op.cc:270] Start()
0627 13:55:40.687284 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536457642 usec, L: 0
0627 13:55:40.687502 (+ 218us) write_op.cc:270] Start()
0627 13:55:40.687527 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536474203 usec, L: 0
0627 13:55:40.701791 (+ 14264us) write_op.cc:270] Start()
0627 13:55:40.701824 (+ 33us) write_op.cc:276] Timestamp: P: 1751032536486369 usec, L: 0
0627 13:55:40.702666 (+ 842us) write_op.cc:270] Start()
0627 13:55:40.702688 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536506618 usec, L: 0
0627 13:55:40.706083 (+ 3395us) write_op.cc:270] Start()
0627 13:55:40.706108 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536519881 usec, L: 0
0627 13:55:40.706294 (+ 186us) write_op.cc:270] Start()
0627 13:55:40.706313 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536533795 usec, L: 0
0627 13:55:40.706489 (+ 176us) write_op.cc:270] Start()
0627 13:55:40.706507 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536582150 usec, L: 0
0627 13:55:40.706678 (+ 171us) write_op.cc:270] Start()
0627 13:55:40.706697 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536583789 usec, L: 0
0627 13:55:40.706979 (+ 282us) write_op.cc:270] Start()
0627 13:55:40.706997 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536612931 usec, L: 0
0627 13:55:40.707139 (+ 142us) write_op.cc:270] Start()
0627 13:55:40.707156 (+ 17us) write_op.cc:276] Timestamp: P: 1751032536614223 usec, L: 0
0627 13:55:40.707359 (+ 203us) write_op.cc:270] Start()
0627 13:55:40.707377 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536637997 usec, L: 0
0627 13:55:40.707538 (+ 161us) write_op.cc:270] Start()
0627 13:55:40.707555 (+ 17us) write_op.cc:276] Timestamp: P: 1751032536644243 usec, L: 0
0627 13:55:40.707700 (+ 145us) write_op.cc:270] Start()
0627 13:55:40.707717 (+ 17us) write_op.cc:276] Timestamp: P: 1751032536658633 usec, L: 0
0627 13:55:40.707902 (+ 185us) write_op.cc:270] Start()
0627 13:55:40.708125 (+ 223us) write_op.cc:276] Timestamp: P: 1751032536680079 usec, L: 0
0627 13:55:40.708906 (+ 781us) write_op.cc:270] Start()
0627 13:55:40.708927 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536705517 usec, L: 0
0627 13:55:40.754468 (+ 45541us) write_op.cc:270] Start()
0627 13:55:40.754497 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536724680 usec, L: 0
0627 13:55:40.755836 (+ 1339us) write_op.cc:270] Start()
0627 13:55:40.755861 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536741421 usec, L: 0
0627 13:55:40.756041 (+ 180us) write_op.cc:270] Start()
0627 13:55:40.756060 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536761211 usec, L: 0
0627 13:55:40.756213 (+ 153us) write_op.cc:270] Start()
0627 13:55:40.756231 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536782948 usec, L: 0
0627 13:55:40.760159 (+ 3928us) write_op.cc:270] Start()
0627 13:55:40.760180 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536807866 usec, L: 0
0627 13:55:40.763854 (+ 3674us) write_op.cc:270] Start()
0627 13:55:40.763880 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536825234 usec, L: 0
0627 13:55:40.765970 (+ 2090us) write_op.cc:270] Start()
0627 13:55:40.765996 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536849631 usec, L: 0
0627 13:55:40.766338 (+ 342us) write_op.cc:270] Start()
0627 13:55:40.768831 (+ 2493us) write_op.cc:276] Timestamp: P: 1751032536852225 usec, L: 0
0627 13:55:40.772816 (+ 3985us) write_op.cc:270] Start()
0627 13:55:40.772840 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536892209 usec, L: 0
0627 13:55:40.776010 (+ 3170us) write_op.cc:270] Start()
0627 13:55:40.776040 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536913144 usec, L: 0
0627 13:55:40.776215 (+ 175us) write_op.cc:270] Start()
0627 13:55:40.776236 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536915102 usec, L: 0
0627 13:55:40.776388 (+ 152us) write_op.cc:270] Start()
0627 13:55:40.776413 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536916020 usec, L: 0
0627 13:55:40.776617 (+ 204us) write_op.cc:270] Start()
0627 13:55:40.776640 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536976528 usec, L: 0
0627 13:55:40.776832 (+ 192us) write_op.cc:270] Start()
0627 13:55:40.776852 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536978150 usec, L: 0
0627 13:55:40.777043 (+ 191us) write_op.cc:270] Start()
0627 13:55:40.777065 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536990426 usec, L: 0
0627 13:55:40.777235 (+ 170us) write_op.cc:270] Start()
0627 13:55:40.777255 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537003800 usec, L: 0
0627 13:55:40.777432 (+ 177us) write_op.cc:270] Start()
0627 13:55:40.777452 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537035505 usec, L: 0
0627 13:55:40.793453 (+ 16001us) write_op.cc:270] Start()
0627 13:55:40.793486 (+ 33us) write_op.cc:276] Timestamp: P: 1751032537052745 usec, L: 0
0627 13:55:40.796350 (+ 2864us) write_op.cc:270] Start()
0627 13:55:40.796375 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537062100 usec, L: 0
0627 13:55:40.796569 (+ 194us) write_op.cc:270] Start()
0627 13:55:40.796589 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537089888 usec, L: 0
0627 13:55:40.818990 (+ 22401us) write_op.cc:270] Start()
0627 13:55:40.819028 (+ 38us) write_op.cc:276] Timestamp: P: 1751032537090615 usec, L: 0
0627 13:55:40.821591 (+ 2563us) write_op.cc:270] Start()
0627 13:55:40.821621 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537205630 usec, L: 0
0627 13:55:40.827811 (+ 6190us) write_op.cc:270] Start()
0627 13:55:40.827839 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537233033 usec, L: 0
0627 13:55:40.864992 (+ 37153us) write_op.cc:270] Start()
0627 13:55:40.865020 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537235670 usec, L: 0
0627 13:55:40.865217 (+ 197us) write_op.cc:270] Start()
0627 13:55:40.866816 (+ 1599us) write_op.cc:276] Timestamp: P: 1751032537247328 usec, L: 0
0627 13:55:40.872212 (+ 5396us) write_op.cc:270] Start()
0627 13:55:40.872237 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537262589 usec, L: 0
0627 13:55:40.880993 (+ 8756us) write_op.cc:270] Start()
0627 13:55:40.881025 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537285408 usec, L: 0
0627 13:55:40.895530 (+ 14505us) write_op.cc:270] Start()
0627 13:55:40.895557 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537299306 usec, L: 0
0627 13:55:40.912002 (+ 16445us) write_op.cc:270] Start()
0627 13:55:40.912034 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537315697 usec, L: 0
0627 13:55:40.916365 (+ 4331us) write_op.cc:270] Start()
0627 13:55:40.916383 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537329513 usec, L: 0
0627 13:55:40.920977 (+ 4594us) write_op.cc:270] Start()
0627 13:55:40.921005 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537360357 usec, L: 0
0627 13:55:40.923161 (+ 2156us) write_op.cc:270] Start()
0627 13:55:40.923186 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537367085 usec, L: 0
0627 13:55:40.924926 (+ 1740us) write_op.cc:270] Start()
0627 13:55:40.924947 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537383007 usec, L: 0
0627 13:55:40.929634 (+ 4687us) write_op.cc:270] Start()
0627 13:55:40.929662 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537424112 usec, L: 0
0627 13:55:40.933614 (+ 3952us) write_op.cc:270] Start()
0627 13:55:40.933646 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537432571 usec, L: 0
0627 13:55:40.933878 (+ 232us) write_op.cc:270] Start()
0627 13:55:40.933902 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537467228 usec, L: 0
0627 13:55:40.947951 (+ 14049us) write_op.cc:270] Start()
0627 13:55:40.947981 (+ 30us) write_op.cc:276] Timestamp: P: 1751032537469017 usec, L: 0
0627 13:55:40.950216 (+ 2235us) write_op.cc:270] Start()
0627 13:55:40.950239 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537478982 usec, L: 0
0627 13:55:40.960377 (+ 10138us) write_op.cc:270] Start()
0627 13:55:40.960404 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537496332 usec, L: 0
0627 13:55:40.963342 (+ 2938us) write_op.cc:270] Start()
0627 13:55:40.963367 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537523837 usec, L: 0
0627 13:55:40.970433 (+ 7066us) write_op.cc:270] Start()
0627 13:55:40.970459 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537531307 usec, L: 0
0627 13:55:40.975278 (+ 4819us) write_op.cc:270] Start()
0627 13:55:40.975300 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537541859 usec, L: 0
0627 13:55:40.977730 (+ 2430us) write_op.cc:270] Start()
0627 13:55:40.977757 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537546207 usec, L: 0
0627 13:55:40.977979 (+ 222us) write_op.cc:270] Start()
0627 13:55:40.978001 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537586444 usec, L: 0
0627 13:55:40.979608 (+ 1607us) write_op.cc:270] Start()
0627 13:55:40.979632 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537587160 usec, L: 0
0627 13:55:40.981377 (+ 1745us) write_op.cc:270] Start()
0627 13:55:40.981402 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537623609 usec, L: 0
0627 13:55:40.990492 (+ 9090us) write_op.cc:270] Start()
0627 13:55:40.990524 (+ 32us) write_op.cc:276] Timestamp: P: 1751032537625595 usec, L: 0
0627 13:55:40.990777 (+ 253us) write_op.cc:270] Start()
0627 13:55:40.990801 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537685990 usec, L: 0
0627 13:55:40.991082 (+ 281us) write_op.cc:270] Start()
0627 13:55:40.991107 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537698159 usec, L: 0
0627 13:55:40.991315 (+ 208us) write_op.cc:270] Start()
0627 13:55:40.991337 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537706239 usec, L: 0
0627 13:55:40.991505 (+ 168us) write_op.cc:270] Start()
0627 13:55:40.991526 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537717333 usec, L: 0
0627 13:55:40.991692 (+ 166us) write_op.cc:270] Start()
0627 13:55:40.991718 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537756138 usec, L: 0
0627 13:55:40.991894 (+ 176us) write_op.cc:270] Start()
0627 13:55:40.991915 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537773928 usec, L: 0
0627 13:55:40.992062 (+ 147us) write_op.cc:270] Start()
0627 13:55:40.992083 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537780610 usec, L: 0
0627 13:55:40.992226 (+ 143us) write_op.cc:270] Start()
0627 13:55:40.992244 (+ 18us) write_op.cc:276] Timestamp: P: 1751032537796816 usec, L: 0
0627 13:55:40.992392 (+ 148us) write_op.cc:270] Start()
0627 13:55:40.992413 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537820861 usec, L: 0
0627 13:55:40.992579 (+ 166us) write_op.cc:270] Start()
0627 13:55:40.992603 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537840327 usec, L: 0
0627 13:55:40.992762 (+ 159us) write_op.cc:270] Start()
0627 13:55:40.992788 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537847664 usec, L: 0
0627 13:55:40.992974 (+ 186us) write_op.cc:270] Start()
0627 13:55:40.992997 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537865199 usec, L: 0
0627 13:55:40.993163 (+ 166us) write_op.cc:270] Start()
0627 13:55:40.993183 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537871572 usec, L: 0
0627 13:55:40.993385 (+ 202us) write_op.cc:270] Start()
0627 13:55:40.993408 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537904845 usec, L: 0
0627 13:55:40.993590 (+ 182us) write_op.cc:270] Start()
0627 13:55:40.993613 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537908768 usec, L: 0
0627 13:55:40.993801 (+ 188us) write_op.cc:270] Start()
0627 13:55:40.993822 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537927745 usec, L: 0
0627 13:55:40.994014 (+ 192us) write_op.cc:270] Start()
0627 13:55:41.018967 (+ 24953us) write_op.cc:276] Timestamp: P: 1751032537929042 usec, L: 0
0627 13:55:41.028862 (+ 9895us) write_op.cc:270] Start()
0627 13:55:41.028889 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537970746 usec, L: 0
0627 13:55:41.031338 (+ 2449us) write_op.cc:270] Start()
0627 13:55:41.031365 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537981194 usec, L: 0
0627 13:55:41.035927 (+ 4562us) write_op.cc:270] Start()
0627 13:55:41.035953 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537983640 usec, L: 0
0627 13:55:41.036149 (+ 196us) write_op.cc:270] Start()
0627 13:55:41.036174 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537987369 usec, L: 0
0627 13:55:41.043158 (+ 6984us) write_op.cc:270] Start()
0627 13:55:41.043216 (+ 58us) write_op.cc:276] Timestamp: P: 1751032538045516 usec, L: 0
0627 13:55:41.043446 (+ 230us) write_op.cc:270] Start()
0627 13:55:41.043470 (+ 24us) write_op.cc:276] Timestamp: P: 1751032538071402 usec, L: 0
0627 13:55:41.043732 (+ 262us) write_op.cc:270] Start()
0627 13:55:41.043754 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538074200 usec, L: 0
0627 13:55:41.059979 (+ 16225us) write_op.cc:270] Start()
0627 13:55:41.060044 (+ 65us) write_op.cc:276] Timestamp: P: 1751032538082962 usec, L: 0
0627 13:55:41.062211 (+ 2167us) write_op.cc:270] Start()
0627 13:55:41.062239 (+ 28us) write_op.cc:276] Timestamp: P: 1751032538102619 usec, L: 0
0627 13:55:41.070806 (+ 8567us) write_op.cc:270] Start()
0627 13:55:41.070846 (+ 40us) write_op.cc:276] Timestamp: P: 1751032538144001 usec, L: 0
0627 13:55:41.075276 (+ 4430us) write_op.cc:270] Start()
0627 13:55:41.075303 (+ 27us) write_op.cc:276] Timestamp: P: 1751032538164619 usec, L: 0
0627 13:55:41.075507 (+ 204us) write_op.cc:270] Start()
0627 13:55:41.075529 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538169998 usec, L: 0
0627 13:55:41.075715 (+ 186us) write_op.cc:270] Start()
0627 13:55:41.075739 (+ 24us) write_op.cc:276] Timestamp: P: 1751032538175412 usec, L: 0
0627 13:55:41.075934 (+ 195us) write_op.cc:270] Start()
0627 13:55:41.075953 (+ 19us) write_op.cc:276] Timestamp: P: 1751032538219862 usec, L: 0
0627 13:55:41.076125 (+ 172us) write_op.cc:270] Start()
0627 13:55:41.076158 (+ 33us) write_op.cc:276] Timestamp: P: 1751032538227123 usec, L: 0
0627 13:55:41.076355 (+ 197us) write_op.cc:270] Start()
0627 13:55:41.076377 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538228445 usec, L: 0
0627 13:55:41.076536 (+ 159us) write_op.cc:270] Start()
0627 13:55:41.076556 (+ 20us) write_op.cc:276] Timestamp: P: 1751032538240745 usec, L: 0
0627 13:55:41.076730 (+ 174us) write_op.cc:270] Start()
0627 13:55:41.076752 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538280047 usec, L: 0
0627 13:55:41.105999 (+ 29247us) write_op.cc:270] Start()
0627 13:55:41.106028 (+ 29us) write_op.cc:276] Timestamp: P: 1751032538289893 usec, L: 0
0627 13:55:41.106270 (+ 242us) write_op.cc:270] Start()
0627 13:55:41.106295 (+ 25us) write_op.cc:276] Timestamp: P: 1751032538291489 usec, L: 0
0627 13:55:41.113127 (+ 6832us) write_op.cc:270] Start()
0627 13:55:41.117551 (+ 4424us) write_op.cc:276] Timestamp: P: 1751032538303842 usec, L: 0
0627 13:55:41.121232 (+ 3681us) write_op.cc:270] Start()
0627 13:55:41.121259 (+ 27us) write_op.cc:276] Timestamp: P: 1751032538349991 usec, L: 0
0627 13:55:41.123738 (+ 2479us) write_op.cc:270] Start()
0627 13:55:41.123765 (+ 27us) write_op.cc:276] Timestamp: P: 1751032538370209 usec, L: 0
0627 13:55:41.131613 (+ 7848us) write_op.cc:270] Start()
0627 13:55:41.131640 (+ 27us) write_op.cc:276] Timestamp: P: 1751032538377042 usec, L: 0
0627 13:55:41.131843 (+ 203us) write_op.cc:270] Start()
0627 13:55:41.131876 (+ 33us) write_op.cc:276] Timestamp: P: 1751032538392239 usec, L: 0
0627 13:55:41.132028 (+ 152us) write_op.cc:270] Start()
0627 13:55:41.132049 (+ 21us) write_op.cc:276] Timestamp: P: 1751032538424440 usec, L: 0
0627 13:55:41.132208 (+ 159us) write_op.cc:270] Start()
0627 13:55:41.132231 (+ 23us) write_op.cc:276] Timestamp: P: 175103
I20250627 13:55:42.601792 9928 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:42.602005 9928 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:42.675928 9928 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:42.676995 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:42.741489 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1022 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:42.772135 9761 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.131433 9919 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.132196 9828 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 7) took 1128 ms. Trace:
I20250627 13:55:43.133023 9919 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.133883 9919 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.134644 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1156 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.145473 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1156 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.146163 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.146720 9919 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.186378 9913 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.187677 9913 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.191526 9829 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 6) took 2283 ms. Trace:
I20250627 13:55:43.208436 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.208460 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.211723 9787 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57736 (request call id 43) took 3726 ms. Trace:
I20250627 13:55:43.211717 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.211867 9787 rpcz_store.cc:276] 0627 13:55:39.484529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:39.484770 (+ 241us) service_pool.cc:225] Handling call
0627 13:55:41.142178 (+1657408us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x616000018154. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:41.459660 (+317482us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x616000017554. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:43.209546 (+1749886us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x616000017b54. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:43.210979 (+ 1433us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8515049088}
I20250627 13:55:43.212080 9919 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.218230 9913 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.488341 9928 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.495048 9928 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.497990 9928 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.498309 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1185 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.498469 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1185 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.507225 9760 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.520900 9928 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.544627 9928 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.564328 9928 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.569913 9928 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.570233 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1156 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.570953 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1156 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.571851 9760 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.579034 9928 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.579286 9913 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.579413 9913 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.580013 9913 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.582449 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.584534 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.589264 9760 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.603216 9913 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.134208 9828 rpcz_store.cc:276] 0627 13:55:42.002777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:42.002843 (+ 66us) service_pool.cc:225] Handling call
0627 13:55:42.018783 (+ 15940us) raft_consensus.cc:1469] Updating replica for 146 ops
0627 13:55:42.135094 (+116311us) raft_consensus.cc:1532] Early marking committed up to index 1010
0627 13:55:42.135168 (+ 74us) raft_consensus.cc:1537] Triggering prepare for 146 ops
0627 13:55:42.140767 (+ 5599us) write_op.cc:270] Start()
0627 13:55:42.173960 (+ 33193us) write_op.cc:276] Timestamp: P: 1751032539258546 usec, L: 0
0627 13:55:42.179349 (+ 5389us) write_op.cc:270] Start()
0627 13:55:42.179381 (+ 32us) write_op.cc:276] Timestamp: P: 1751032539276087 usec, L: 0
0627 13:55:42.179574 (+ 193us) write_op.cc:270] Start()
0627 13:55:42.179597 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539334213 usec, L: 0
0627 13:55:42.219185 (+ 39588us) write_op.cc:270] Start()
0627 13:55:42.219215 (+ 30us) write_op.cc:276] Timestamp: P: 1751032539360923 usec, L: 0
0627 13:55:42.224057 (+ 4842us) write_op.cc:270] Start()
0627 13:55:42.224080 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539361665 usec, L: 0
0627 13:55:42.240042 (+ 15962us) write_op.cc:270] Start()
0627 13:55:42.240068 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539363237 usec, L: 0
0627 13:55:42.243244 (+ 3176us) write_op.cc:270] Start()
0627 13:55:42.243300 (+ 56us) write_op.cc:276] Timestamp: P: 1751032539389241 usec, L: 0
0627 13:55:42.243472 (+ 172us) write_op.cc:270] Start()
0627 13:55:42.243494 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539392034 usec, L: 0
0627 13:55:42.243664 (+ 170us) write_op.cc:270] Start()
0627 13:55:42.243685 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539449871 usec, L: 0
0627 13:55:42.243843 (+ 158us) write_op.cc:270] Start()
0627 13:55:42.243865 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539450771 usec, L: 0
0627 13:55:42.244023 (+ 158us) write_op.cc:270] Start()
0627 13:55:42.244044 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539456910 usec, L: 0
0627 13:55:42.244197 (+ 153us) write_op.cc:270] Start()
0627 13:55:42.244219 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539458966 usec, L: 0
0627 13:55:42.244365 (+ 146us) write_op.cc:270] Start()
0627 13:55:42.244387 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539491453 usec, L: 0
0627 13:55:42.244539 (+ 152us) write_op.cc:270] Start()
0627 13:55:42.244573 (+ 34us) write_op.cc:276] Timestamp: P: 1751032539505929 usec, L: 0
0627 13:55:42.244732 (+ 159us) write_op.cc:270] Start()
0627 13:55:42.244752 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539514530 usec, L: 0
0627 13:55:42.244902 (+ 150us) write_op.cc:270] Start()
0627 13:55:42.244923 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539532461 usec, L: 0
0627 13:55:42.245070 (+ 147us) write_op.cc:270] Start()
0627 13:55:42.245090 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539556902 usec, L: 0
0627 13:55:42.290381 (+ 45291us) write_op.cc:270] Start()
0627 13:55:42.290433 (+ 52us) write_op.cc:276] Timestamp: P: 1751032539571566 usec, L: 0
0627 13:55:42.290700 (+ 267us) write_op.cc:270] Start()
0627 13:55:42.290731 (+ 31us) write_op.cc:276] Timestamp: P: 1751032539575657 usec, L: 0
0627 13:55:42.290936 (+ 205us) write_op.cc:270] Start()
0627 13:55:42.290964 (+ 28us) write_op.cc:276] Timestamp: P: 1751032539609622 usec, L: 0
0627 13:55:42.291644 (+ 680us) write_op.cc:270] Start()
0627 13:55:42.291667 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539627780 usec, L: 0
0627 13:55:42.291816 (+ 149us) write_op.cc:270] Start()
0627 13:55:42.291840 (+ 24us) write_op.cc:276] Timestamp: P: 1751032539628898 usec, L: 0
0627 13:55:42.297390 (+ 5550us) write_op.cc:270] Start()
0627 13:55:42.297416 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539633520 usec, L: 0
0627 13:55:42.298599 (+ 1183us) write_op.cc:270] Start()
0627 13:55:42.298619 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539689553 usec, L: 0
0627 13:55:42.302085 (+ 3466us) write_op.cc:270] Start()
0627 13:55:42.302117 (+ 32us) write_op.cc:276] Timestamp: P: 1751032539698042 usec, L: 0
0627 13:55:42.302315 (+ 198us) write_op.cc:270] Start()
0627 13:55:42.302342 (+ 27us) write_op.cc:276] Timestamp: P: 1751032539706662 usec, L: 0
0627 13:55:42.302922 (+ 580us) write_op.cc:270] Start()
0627 13:55:42.302954 (+ 32us) write_op.cc:276] Timestamp: P: 1751032539734717 usec, L: 0
0627 13:55:42.303160 (+ 206us) write_op.cc:270] Start()
0627 13:55:42.303185 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539748388 usec, L: 0
0627 13:55:42.303368 (+ 183us) write_op.cc:270] Start()
0627 13:55:42.303394 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539757291 usec, L: 0
0627 13:55:42.303551 (+ 157us) write_op.cc:270] Start()
0627 13:55:42.303585 (+ 34us) write_op.cc:276] Timestamp: P: 1751032539783408 usec, L: 0
0627 13:55:42.303873 (+ 288us) write_op.cc:270] Start()
0627 13:55:42.303900 (+ 27us) write_op.cc:276] Timestamp: P: 1751032539801739 usec, L: 0
0627 13:55:42.304128 (+ 228us) write_op.cc:270] Start()
0627 13:55:42.304150 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539802431 usec, L: 0
0627 13:55:42.304335 (+ 185us) write_op.cc:270] Start()
0627 13:55:42.304358 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539809425 usec, L: 0
0627 13:55:42.304609 (+ 251us) write_op.cc:270] Start()
0627 13:55:42.304636 (+ 27us) write_op.cc:276] Timestamp: P: 1751032539836042 usec, L: 0
0627 13:55:42.304768 (+ 132us) write_op.cc:270] Start()
0627 13:55:42.304791 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539876400 usec, L: 0
0627 13:55:42.304946 (+ 155us) write_op.cc:270] Start()
0627 13:55:42.304966 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539889536 usec, L: 0
0627 13:55:42.305099 (+ 133us) write_op.cc:270] Start()
0627 13:55:42.305118 (+ 19us) write_op.cc:276] Timestamp: P: 1751032539913800 usec, L: 0
0627 13:55:42.305240 (+ 122us) write_op.cc:270] Start()
0627 13:55:42.305258 (+ 18us) write_op.cc:276] Timestamp: P: 1751032539921370 usec, L: 0
0627 13:55:42.320966 (+ 15708us) write_op.cc:270] Start()
0627 13:55:42.320991 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539935848 usec, L: 0
0627 13:55:42.330735 (+ 9744us) write_op.cc:270] Start()
0627 13:55:42.330760 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539957209 usec, L: 0
0627 13:55:42.335282 (+ 4522us) write_op.cc:270] Start()
0627 13:55:42.335307 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540007031 usec, L: 0
0627 13:55:42.362909 (+ 27602us) write_op.cc:270] Start()
0627 13:55:42.362938 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540007960 usec, L: 0
0627 13:55:42.363124 (+ 186us) write_op.cc:270] Start()
0627 13:55:42.363146 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540010215 usec, L: 0
0627 13:55:42.364280 (+ 1134us) write_op.cc:270] Start()
0627 13:55:42.364307 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540027413 usec, L: 0
0627 13:55:42.364493 (+ 186us) write_op.cc:270] Start()
0627 13:55:42.364514 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540062419 usec, L: 0
0627 13:55:42.364656 (+ 142us) write_op.cc:270] Start()
0627 13:55:42.364685 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540085975 usec, L: 0
0627 13:55:42.389132 (+ 24447us) write_op.cc:270] Start()
0627 13:55:42.389164 (+ 32us) write_op.cc:276] Timestamp: P: 1751032540088449 usec, L: 0
0627 13:55:42.389355 (+ 191us) write_op.cc:270] Start()
0627 13:55:42.389375 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540110097 usec, L: 0
0627 13:55:42.395085 (+ 5710us) write_op.cc:270] Start()
0627 13:55:42.395130 (+ 45us) write_op.cc:276] Timestamp: P: 1751032540143433 usec, L: 0
0627 13:55:42.395914 (+ 784us) write_op.cc:270] Start()
0627 13:55:42.395940 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540152869 usec, L: 0
0627 13:55:42.396129 (+ 189us) write_op.cc:270] Start()
0627 13:55:42.396151 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540159503 usec, L: 0
0627 13:55:42.419995 (+ 23844us) write_op.cc:270] Start()
0627 13:55:42.420029 (+ 34us) write_op.cc:276] Timestamp: P: 1751032540183546 usec, L: 0
0627 13:55:42.434939 (+ 14910us) write_op.cc:270] Start()
0627 13:55:42.434980 (+ 41us) write_op.cc:276] Timestamp: P: 1751032540231335 usec, L: 0
0627 13:55:42.436130 (+ 1150us) write_op.cc:270] Start()
0627 13:55:42.436165 (+ 35us) write_op.cc:276] Timestamp: P: 1751032540247707 usec, L: 0
0627 13:55:42.437951 (+ 1786us) write_op.cc:270] Start()
0627 13:55:42.437980 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540269496 usec, L: 0
0627 13:55:42.438715 (+ 735us) write_op.cc:270] Start()
0627 13:55:42.438741 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540290129 usec, L: 0
0627 13:55:42.454413 (+ 15672us) write_op.cc:270] Start()
0627 13:55:42.454447 (+ 34us) write_op.cc:276] Timestamp: P: 1751032540290762 usec, L: 0
0627 13:55:42.478946 (+ 24499us) write_op.cc:270] Start()
0627 13:55:42.478989 (+ 43us) write_op.cc:276] Timestamp: P: 1751032540313880 usec, L: 0
0627 13:55:42.479243 (+ 254us) write_op.cc:270] Start()
0627 13:55:42.479269 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540348290 usec, L: 0
0627 13:55:42.502744 (+ 23475us) write_op.cc:270] Start()
0627 13:55:42.512945 (+ 10201us) write_op.cc:276] Timestamp: P: 1751032540354432 usec, L: 0
0627 13:55:42.528864 (+ 15919us) write_op.cc:270] Start()
0627 13:55:42.528894 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540360407 usec, L: 0
0627 13:55:42.553562 (+ 24668us) write_op.cc:270] Start()
0627 13:55:42.553595 (+ 33us) write_op.cc:276] Timestamp: P: 1751032540378226 usec, L: 0
0627 13:55:42.567399 (+ 13804us) write_op.cc:270] Start()
0627 13:55:42.567435 (+ 36us) write_op.cc:276] Timestamp: P: 1751032540410037 usec, L: 0
0627 13:55:42.567650 (+ 215us) write_op.cc:270] Start()
0627 13:55:42.567675 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540413975 usec, L: 0
0627 13:55:42.589615 (+ 21940us) write_op.cc:270] Start()
0627 13:55:42.589646 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540422798 usec, L: 0
0627 13:55:42.596307 (+ 6661us) write_op.cc:270] Start()
0627 13:55:42.596343 (+ 36us) write_op.cc:276] Timestamp: P: 1751032540444142 usec, L: 0
0627 13:55:42.596563 (+ 220us) write_op.cc:270] Start()
0627 13:55:42.596589 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540475995 usec, L: 0
0627 13:55:42.602903 (+ 6314us) write_op.cc:270] Start()
0627 13:55:42.602932 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540486649 usec, L: 0
0627 13:55:42.611122 (+ 8190us) write_op.cc:270] Start()
0627 13:55:42.611159 (+ 37us) write_op.cc:276] Timestamp: P: 1751032540487802 usec, L: 0
0627 13:55:42.611380 (+ 221us) write_op.cc:270] Start()
0627 13:55:42.611399 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540504089 usec, L: 0
0627 13:55:42.611531 (+ 132us) write_op.cc:270] Start()
0627 13:55:42.611552 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540560852 usec, L: 0
0627 13:55:42.611698 (+ 146us) write_op.cc:270] Start()
0627 13:55:42.611725 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540563020 usec, L: 0
0627 13:55:42.611890 (+ 165us) write_op.cc:270] Start()
0627 13:55:42.611916 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540582202 usec, L: 0
0627 13:55:42.612103 (+ 187us) write_op.cc:270] Start()
0627 13:55:42.612126 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540598973 usec, L: 0
0627 13:55:42.614205 (+ 2079us) write_op.cc:270] Start()
0627 13:55:42.614233 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540616294 usec, L: 0
0627 13:55:42.614381 (+ 148us) write_op.cc:270] Start()
0627 13:55:42.614399 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540652210 usec, L: 0
0627 13:55:42.614520 (+ 121us) write_op.cc:270] Start()
0627 13:55:42.614536 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540656119 usec, L: 0
0627 13:55:42.614687 (+ 151us) write_op.cc:270] Start()
0627 13:55:42.614708 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540681880 usec, L: 0
0627 13:55:42.614837 (+ 129us) write_op.cc:270] Start()
0627 13:55:42.614852 (+ 15us) write_op.cc:276] Timestamp: P: 1751032540693052 usec, L: 0
0627 13:55:42.615000 (+ 148us) write_op.cc:270] Start()
0627 13:55:42.615025 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540719241 usec, L: 0
0627 13:55:42.615264 (+ 239us) write_op.cc:270] Start()
0627 13:55:42.615294 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540732183 usec, L: 0
0627 13:55:42.615503 (+ 209us) write_op.cc:270] Start()
0627 13:55:42.615528 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540771301 usec, L: 0
0627 13:55:42.655887 (+ 40359us) write_op.cc:270] Start()
0627 13:55:42.655923 (+ 36us) write_op.cc:276] Timestamp: P: 1751032540774324 usec, L: 0
0627 13:55:42.662570 (+ 6647us) write_op.cc:270] Start()
0627 13:55:42.662595 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540817692 usec, L: 0
0627 13:55:42.663460 (+ 865us) write_op.cc:270] Start()
0627 13:55:42.663497 (+ 37us) write_op.cc:276] Timestamp: P: 1751032540819253 usec, L: 0
0627 13:55:42.663708 (+ 211us) write_op.cc:270] Start()
0627 13:55:42.663726 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540840367 usec, L: 0
0627 13:55:42.675674 (+ 11948us) write_op.cc:270] Start()
0627 13:55:42.675703 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540848543 usec, L: 0
0627 13:55:42.679471 (+ 3768us) write_op.cc:270] Start()
0627 13:55:42.679502 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540862091 usec, L: 0
0627 13:55:42.683645 (+ 4143us) write_op.cc:270] Start()
0627 13:55:42.683674 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540884389 usec, L: 0
0627 13:55:42.688463 (+ 4789us) write_op.cc:270] Start()
0627 13:55:42.688488 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540898411 usec, L: 0
0627 13:55:42.688669 (+ 181us) write_op.cc:270] Start()
0627 13:55:42.688693 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540926527 usec, L: 0
0627 13:55:42.703672 (+ 14979us) write_op.cc:270] Start()
0627 13:55:42.703704 (+ 32us) write_op.cc:276] Timestamp: P: 1751032540930787 usec, L: 0
0627 13:55:42.703894 (+ 190us) write_op.cc:270] Start()
0627 13:55:42.703915 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540934415 usec, L: 0
0627 13:55:42.718010 (+ 14095us) write_op.cc:270] Start()
0627 13:55:42.718070 (+ 60us) write_op.cc:276] Timestamp: P: 1751032540955843 usec, L: 0
0627 13:55:42.727379 (+ 9309us) write_op.cc:270] Start()
0627 13:55:42.727413 (+ 34us) write_op.cc:276] Timestamp: P: 1751032540989334 usec, L: 0
0627 13:55:42.731895 (+ 4482us) write_op.cc:270] Start()
0627 13:55:42.731923 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541010322 usec, L: 0
0627 13:55:42.732149 (+ 226us) write_op.cc:270] Start()
0627 13:55:42.732174 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541016178 usec, L: 0
0627 13:55:42.732336 (+ 162us) write_op.cc:270] Start()
0627 13:55:42.732353 (+ 17us) write_op.cc:276] Timestamp: P: 1751032541029517 usec, L: 0
0627 13:55:42.732493 (+ 140us) write_op.cc:270] Start()
0627 13:55:42.732513 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541061775 usec, L: 0
0627 13:55:42.732695 (+ 182us) write_op.cc:270] Start()
0627 13:55:42.732718 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541063075 usec, L: 0
0627 13:55:42.732879 (+ 161us) write_op.cc:270] Start()
0627 13:55:42.732899 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541074219 usec, L: 0
0627 13:55:42.733038 (+ 139us) write_op.cc:270] Start()
0627 13:55:42.733057 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541102352 usec, L: 0
0627 13:55:42.733230 (+ 173us) write_op.cc:270] Start()
0627 13:55:42.733251 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541147322 usec, L: 0
0627 13:55:42.733402 (+ 151us) write_op.cc:270] Start()
0627 13:55:42.733423 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541160299 usec, L: 0
0627 13:55:42.733566 (+ 143us) write_op.cc:270] Start()
0627 13:55:42.733589 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541168164 usec, L: 0
0627 13:55:42.733756 (+ 167us) write_op.cc:270] Start()
0627 13:55:42.733779 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541190499 usec, L: 0
0627 13:55:42.733910 (+ 131us) write_op.cc:270] Start()
0627 13:55:42.733930 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541210004 usec, L: 0
0627 13:55:42.737539 (+ 3609us) write_op.cc:270] Start()
0627 13:55:42.737565 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541225733 usec, L: 0
0627 13:55:42.778001 (+ 40436us) write_op.cc:270] Start()
0627 13:55:42.778049 (+ 48us) write_op.cc:276] Timestamp: P: 1751032541247208 usec, L: 0
0627 13:55:42.807234 (+ 29185us) write_op.cc:270] Start()
0627 13:55:42.807272 (+ 38us) write_op.cc:276] Timestamp: P: 1751032541295469 usec, L: 0
0627 13:55:42.807520 (+ 248us) write_op.cc:270] Start()
0627 13:55:42.807549 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541306189 usec, L: 0
0627 13:55:42.807758 (+ 209us) write_op.cc:270] Start()
0627 13:55:42.807782 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541311158 usec, L: 0
0627 13:55:42.819034 (+ 11252us) write_op.cc:270] Start()
0627 13:55:42.819074 (+ 40us) write_op.cc:276] Timestamp: P: 1751032541321700 usec, L: 0
0627 13:55:42.826921 (+ 7847us) write_op.cc:270] Start()
0627 13:55:42.826961 (+ 40us) write_op.cc:276] Timestamp: P: 1751032541334844 usec, L: 0
0627 13:55:42.829542 (+ 2581us) write_op.cc:270] Start()
0627 13:55:42.829566 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541357818 usec, L: 0
0627 13:55:42.842264 (+ 12698us) write_op.cc:270] Start()
0627 13:55:42.842295 (+ 31us) write_op.cc:276] Timestamp: P: 1751032541407432 usec, L: 0
0627 13:55:42.842491 (+ 196us) write_op.cc:270] Start()
0627 13:55:42.842516 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541410695 usec, L: 0
0627 13:55:42.842650 (+ 134us) write_op.cc:270] Start()
0627 13:55:42.842670 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541413104 usec, L: 0
0627 13:55:42.842834 (+ 164us) write_op.cc:270] Start()
0627 13:55:42.842857 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541435678 usec, L: 0
0627 13:55:42.855654 (+ 12797us) write_op.cc:270] Start()
0627 13:55:42.855683 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541483288 usec, L: 0
0627 13:55:42.861294 (+ 5611us) write_op.cc:270] Start()
0627 13:55:42.861324 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541493521 usec, L: 0
0627 13:55:42.874467 (+ 13143us) write_op.cc:270] Start()
0627 13:55:42.874492 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541494139 usec, L: 0
0627 13:55:42.879438 (+ 4946us) write_op.cc:270] Start()
0627 13:55:42.879463 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541504851 usec, L: 0
0627 13:55:42.884002 (+ 4539us) write_op.cc:270] Start()
0627 13:55:42.884026 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541546655 usec, L: 0
0627 13:55:42.884202 (+ 176us) write_op.cc:270] Start()
0627 13:55:42.884221 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541575118 usec, L: 0
0627 13:55:42.884388 (+ 167us) write_op.cc:270] Start()
0627 13:55:42.884408 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541582078 usec, L: 0
0627 13:55:42.891351 (+ 6943us) write_op.cc:270] Start()
0627 13:55:42.891378 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541589358 usec, L: 0
0627 13:55:42.891554 (+ 176us) write_op.cc:270] Start()
0627 13:55:42.891577 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541602388 usec, L: 0
0627 13:55:42.891774 (+ 197us) write_op.cc:270] Start()
0627 13:55:42.891796 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541643161 usec, L: 0
0627 13:55:42.891942 (+ 146us) write_op.cc:270] Start()
0627 13:55:42.891961 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541644285 usec, L: 0
0627 13:55:42.896392 (+ 4431us) write_op.cc:270] Start()
0627 13:55:42.896452 (+ 60us) write_op.cc:276] Timestamp: P: 1751032541675361 usec, L: 0
0627 13:55:42.896635 (+ 183us) write_op.cc:270] Start()
0627 13:55:42.896654 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541696589 usec, L: 0
0627 13:55:42.896810 (+ 156us) write_op.cc:270] Start()
0627 13:55:42.896836 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541699471 usec, L: 0
0627 13:55:42.897058 (+ 222us) write_op.cc:270] Start()
0627 13:55:42.897077 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541701917 usec, L: 0
0627 13:55:42.897252 (+ 175us) write_op.cc:270] Start()
0627 13:55:42.897275 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541755391 usec, L: 0
0627 13:55:42.897387 (+ 112us) write_op.cc:270] Start()
0627 13:55:42.897401 (+ 14us) write_op.cc:276] Timestamp: P: 1751032541791694 usec, L: 0
0627 13:55:42.897483 (+ 82us) write_op.cc:270] Start()
0627 13:55:42.897495 (+ 12us) write_op.cc:276] Timestamp: P: 1751032541799143 usec, L: 0
0627 13:55:42.897583 (+ 88us) write_op.cc:270] Start()
0627 13:55:42.897595 (+ 12us) write_op.cc:276] Timestamp: P: 1751032541807403 usec, L: 0
0627 13:55:42.897684 (+ 89us) write_op.cc:270] Start()
0627 13:55:42.897703 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541847158 usec, L: 0
0627 13:55:42.897870 (+ 167us) write_op.cc:270] Start()
0627 13:55:42.897892 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541873308 usec, L: 0
0627 13:55:42.898078 (+ 186us) write_op.cc:270] Start()
0627 13:55:42.898100 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541876493 usec, L: 0
0627 13:55:42.898297 (+ 197us) write_op.cc:270] Start()
0627 13:55:42.898319 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541896536 usec, L: 0
0627 13:55:42.898489 (+ 170us) write_op.cc:270] Start()
0627 13:55:42.898510 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541929723 usec, L: 0
0627 13:55:42.898655 (+ 145us) write_op.cc:270] Start()
0627 13:55:42.898675 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541941464 usec, L: 0
0627 13:55:42.898824 (+ 149us) write_op.cc:270] Start()
0627 13:55:42.902086 (+ 3262us) write_op.cc:276] Timestamp: P: 1751032541948472 usec, L: 0
0627 13:55:42.915079 (+ 12993us) write_op.cc:270] Start()
0627 13:55:42.915104 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541964897 usec, L: 0
0627 13:55:43.008430 (+ 93326us) log.cc:844] Serialized 98208 byte log entry
0627 13:55:43.008588 (+ 158us) raft_consensus.cc:1643] Marking committed up to 1156
0627 13:55:43.131317 (+122729us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x6160000d27b8. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243eea03d0 00007f243ef595f7 00007f243ef55507 00007f2442cb56ed 00007f243d17f3b6 00007f243d17f0e5 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf
0627 13:55:43.131329 (+ 12us) raft_consensus.cc:1695] Filling consensus response to leader.
0627 13:55:43.131370 (+ 41us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0627 13:55:43.131423 (+ 53us) raft_consensus.cc:1683] finished
0627 13:55:43.131425 (+ 2us) raft_consensus.cc:1689] UpdateReplicas() finished
0627 13:55:43.131501 (+ 76us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0627 13:55:42.710470 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.721087 (+ 10617us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.721093 (+ 6us) write_op.cc:435] Acquired schema lock
0627 13:55:42.721095 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.721233 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.721377 (+ 144us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.721385 (+ 8us) tablet.cc:683] Acquiring locks for 6 operations
0627 13:55:42.728589 (+ 7204us) tablet.cc:699] Row locks acquired
0627 13:55:42.728594 (+ 5us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008634 (+280040us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.728690 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.766867 (+ 38177us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.766874 (+ 7us) write_op.cc:435] Acquired schema lock
0627 13:55:42.766878 (+ 4us) tablet.cc:660] Decoding operations
0627 13:55:42.767058 (+ 180us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.767070 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.767075 (+ 5us) tablet.cc:683] Acquiring locks for 13 operations
0627 13:55:42.786851 (+ 19776us) tablet.cc:699] Row locks acquired
0627 13:55:42.786854 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008681 (+221827us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.786969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.787068 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.787073 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.787075 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.787170 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.787182 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.787185 (+ 3us) tablet.cc:683] Acquiring locks for 4 operations
0627 13:55:42.798407 (+ 11222us) tablet.cc:699] Row locks acquired
0627 13:55:42.798410 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008712 (+210302us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.798507 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.798597 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.798602 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.798604 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.839617 (+ 41013us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.839635 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.839640 (+ 5us) tablet.cc:683] Acquiring locks for 11 operations
0627 13:55:42.839758 (+ 118us) tablet.cc:699] Row locks acquired
0627 13:55:42.839761 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008740 (+168979us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.839863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.839955 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.839960 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.839963 (+ 3us) tablet.cc:660] Decoding operations
0627 13:55:42.840135 (+ 172us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.840149 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.840153 (+ 4us) tablet.cc:683] Acquiring locks for 9 operations
0627 13:55:42.840251 (+ 98us) tablet.cc:699] Row locks acquired
0627 13:55:42.840254 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008766 (+168512us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.840351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.840430 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.840435 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.840437 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.840563 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.840573 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.840581 (+ 8us) tablet.cc:683] Acquiring locks for 9 operations
0627 13:55:42.840660 (+ 79us) tablet.cc:699] Row locks acquired
0627 13:55:42.840662 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008792 (+168130us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.840785 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.840871 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.840876 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.840878 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.840994 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.841006 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.841015 (+ 9us) tablet.cc:683] Acquiring locks for 7 operations
0627 13:55:42.841080 (+ 65us) tablet.cc:699] Row locks acquired
0627 13:55:42.841082 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008828 (+167746us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.841190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82aebc7eff8c4c8a86eb994bf24bd99c
0627 13:55:42.841357 (+ 167us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.841362 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.841364 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.841474 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.841486 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.841490 (+ 4us) tablet.cc:683] Acquiring locks for 6 operations
0627 13:55:42.841581 (+ 91us) tablet.cc:699] Row locks acquired
0627 13:55:42.841583 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.008877 (+167294us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.841685 (+ 0us) write_
I20250627 13:55:43.831616 9929 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.851087 9830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 35) took 1356 ms. Trace:
I20250627 13:55:43.879829 9929 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:43.889386 9929 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:43.905418 9929 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:43.906080 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1235 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:43.907054 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c370dad30118471cbdc7094f7209aad5" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1235 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:43.907591 9761 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:43.908026 9929 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.862075 9830 rpcz_store.cc:276] 0627 13:55:42.474646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:42.498419 (+ 23773us) service_pool.cc:225] Handling call
0627 13:55:42.700714 (+202295us) raft_consensus.cc:1469] Updating replica for 159 ops
0627 13:55:42.832677 (+131963us) raft_consensus.cc:1532] Early marking committed up to index 1022
0627 13:55:42.832766 (+ 89us) raft_consensus.cc:1537] Triggering prepare for 159 ops
0627 13:55:42.832897 (+ 131us) write_op.cc:270] Start()
0627 13:55:42.832930 (+ 33us) write_op.cc:276] Timestamp: P: 1751032539498119 usec, L: 0
0627 13:55:42.833161 (+ 231us) write_op.cc:270] Start()
0627 13:55:42.833185 (+ 24us) write_op.cc:276] Timestamp: P: 1751032539516262 usec, L: 0
0627 13:55:42.833383 (+ 198us) write_op.cc:270] Start()
0627 13:55:42.833405 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539517322 usec, L: 0
0627 13:55:42.833605 (+ 200us) write_op.cc:270] Start()
0627 13:55:42.833628 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539535155 usec, L: 0
0627 13:55:42.833789 (+ 161us) write_op.cc:270] Start()
0627 13:55:42.833809 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539551246 usec, L: 0
0627 13:55:42.833970 (+ 161us) write_op.cc:270] Start()
0627 13:55:42.833993 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539562195 usec, L: 0
0627 13:55:42.834178 (+ 185us) write_op.cc:270] Start()
0627 13:55:42.834216 (+ 38us) write_op.cc:276] Timestamp: P: 1751032539569759 usec, L: 0
0627 13:55:42.866221 (+ 32005us) write_op.cc:270] Start()
0627 13:55:42.866250 (+ 29us) write_op.cc:276] Timestamp: P: 1751032539621196 usec, L: 0
0627 13:55:42.866461 (+ 211us) write_op.cc:270] Start()
0627 13:55:42.866487 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539624496 usec, L: 0
0627 13:55:42.866649 (+ 162us) write_op.cc:270] Start()
0627 13:55:42.866673 (+ 24us) write_op.cc:276] Timestamp: P: 1751032539625652 usec, L: 0
0627 13:55:42.866838 (+ 165us) write_op.cc:270] Start()
0627 13:55:42.866859 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539628129 usec, L: 0
0627 13:55:42.873083 (+ 6224us) write_op.cc:270] Start()
0627 13:55:42.873109 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539674647 usec, L: 0
0627 13:55:42.885236 (+ 12127us) write_op.cc:270] Start()
0627 13:55:42.885265 (+ 29us) write_op.cc:276] Timestamp: P: 1751032539692650 usec, L: 0
0627 13:55:42.892256 (+ 6991us) write_op.cc:270] Start()
0627 13:55:42.892282 (+ 26us) write_op.cc:276] Timestamp: P: 1751032539695333 usec, L: 0
0627 13:55:42.899546 (+ 7264us) write_op.cc:270] Start()
0627 13:55:42.899574 (+ 28us) write_op.cc:276] Timestamp: P: 1751032539705104 usec, L: 0
0627 13:55:42.899743 (+ 169us) write_op.cc:270] Start()
0627 13:55:42.899767 (+ 24us) write_op.cc:276] Timestamp: P: 1751032539732865 usec, L: 0
0627 13:55:42.899927 (+ 160us) write_op.cc:270] Start()
0627 13:55:42.899949 (+ 22us) write_op.cc:276] Timestamp: P: 1751032539759071 usec, L: 0
0627 13:55:42.900132 (+ 183us) write_op.cc:270] Start()
0627 13:55:42.900150 (+ 18us) write_op.cc:276] Timestamp: P: 1751032539780858 usec, L: 0
0627 13:55:42.900302 (+ 152us) write_op.cc:270] Start()
0627 13:55:42.900322 (+ 20us) write_op.cc:276] Timestamp: P: 1751032539805092 usec, L: 0
0627 13:55:42.900488 (+ 166us) write_op.cc:270] Start()
0627 13:55:42.900509 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539805818 usec, L: 0
0627 13:55:42.909937 (+ 9428us) write_op.cc:270] Start()
0627 13:55:42.909978 (+ 41us) write_op.cc:276] Timestamp: P: 1751032539822707 usec, L: 0
0627 13:55:42.910203 (+ 225us) write_op.cc:270] Start()
0627 13:55:42.910228 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539849034 usec, L: 0
0627 13:55:42.914777 (+ 4549us) write_op.cc:270] Start()
0627 13:55:42.914800 (+ 23us) write_op.cc:276] Timestamp: P: 1751032539889108 usec, L: 0
0627 13:55:42.919320 (+ 4520us) write_op.cc:270] Start()
0627 13:55:42.919345 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539890999 usec, L: 0
0627 13:55:42.930713 (+ 11368us) write_op.cc:270] Start()
0627 13:55:42.930743 (+ 30us) write_op.cc:276] Timestamp: P: 1751032539900417 usec, L: 0
0627 13:55:42.933899 (+ 3156us) write_op.cc:270] Start()
0627 13:55:42.933928 (+ 29us) write_op.cc:276] Timestamp: P: 1751032539911882 usec, L: 0
0627 13:55:42.934171 (+ 243us) write_op.cc:270] Start()
0627 13:55:42.934196 (+ 25us) write_op.cc:276] Timestamp: P: 1751032539938250 usec, L: 0
0627 13:55:42.939532 (+ 5336us) write_op.cc:270] Start()
0627 13:55:42.939553 (+ 21us) write_op.cc:276] Timestamp: P: 1751032539957196 usec, L: 0
0627 13:55:42.942170 (+ 2617us) write_op.cc:270] Start()
0627 13:55:42.942191 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540004533 usec, L: 0
0627 13:55:42.943069 (+ 878us) write_op.cc:270] Start()
0627 13:55:42.943091 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540005199 usec, L: 0
0627 13:55:42.943557 (+ 466us) write_op.cc:270] Start()
0627 13:55:42.943583 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540024314 usec, L: 0
0627 13:55:42.943770 (+ 187us) write_op.cc:270] Start()
0627 13:55:42.943793 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540031315 usec, L: 0
0627 13:55:42.944707 (+ 914us) write_op.cc:270] Start()
0627 13:55:42.944730 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540065355 usec, L: 0
0627 13:55:42.961263 (+ 16533us) write_op.cc:270] Start()
0627 13:55:42.961296 (+ 33us) write_op.cc:276] Timestamp: P: 1751032540094542 usec, L: 0
0627 13:55:42.961506 (+ 210us) write_op.cc:270] Start()
0627 13:55:42.961530 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540095697 usec, L: 0
0627 13:55:42.961679 (+ 149us) write_op.cc:270] Start()
0627 13:55:42.961698 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540126895 usec, L: 0
0627 13:55:42.961893 (+ 195us) write_op.cc:270] Start()
0627 13:55:42.961916 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540162235 usec, L: 0
0627 13:55:42.962124 (+ 208us) write_op.cc:270] Start()
0627 13:55:42.962147 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540163244 usec, L: 0
0627 13:55:42.962331 (+ 184us) write_op.cc:270] Start()
0627 13:55:42.962353 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540163945 usec, L: 0
0627 13:55:42.962496 (+ 143us) write_op.cc:270] Start()
0627 13:55:42.962519 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540186371 usec, L: 0
0627 13:55:42.962713 (+ 194us) write_op.cc:270] Start()
0627 13:55:42.962737 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540261990 usec, L: 0
0627 13:55:42.962921 (+ 184us) write_op.cc:270] Start()
0627 13:55:42.962945 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540262619 usec, L: 0
0627 13:55:42.963133 (+ 188us) write_op.cc:270] Start()
0627 13:55:42.963155 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540279458 usec, L: 0
0627 13:55:42.963296 (+ 141us) write_op.cc:270] Start()
0627 13:55:42.963317 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540288233 usec, L: 0
0627 13:55:42.963464 (+ 147us) write_op.cc:270] Start()
0627 13:55:42.963485 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540296420 usec, L: 0
0627 13:55:42.989001 (+ 25516us) write_op.cc:270] Start()
0627 13:55:42.989026 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540316492 usec, L: 0
0627 13:55:42.989509 (+ 483us) write_op.cc:270] Start()
0627 13:55:42.989531 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540343136 usec, L: 0
0627 13:55:42.999619 (+ 10088us) write_op.cc:270] Start()
0627 13:55:42.999644 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540350925 usec, L: 0
0627 13:55:43.001778 (+ 2134us) write_op.cc:270] Start()
0627 13:55:43.001802 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540361913 usec, L: 0
0627 13:55:43.007964 (+ 6162us) write_op.cc:270] Start()
0627 13:55:43.007991 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540381124 usec, L: 0
0627 13:55:43.017335 (+ 9344us) write_op.cc:270] Start()
0627 13:55:43.017363 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540409900 usec, L: 0
0627 13:55:43.024126 (+ 6763us) write_op.cc:270] Start()
0627 13:55:43.024154 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540417268 usec, L: 0
0627 13:55:43.048685 (+ 24531us) write_op.cc:270] Start()
0627 13:55:43.048714 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540424553 usec, L: 0
0627 13:55:43.056548 (+ 7834us) write_op.cc:270] Start()
0627 13:55:43.056572 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540447019 usec, L: 0
0627 13:55:43.075053 (+ 18481us) write_op.cc:270] Start()
0627 13:55:43.075080 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540477742 usec, L: 0
0627 13:55:43.081431 (+ 6351us) write_op.cc:270] Start()
0627 13:55:43.081456 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540491814 usec, L: 0
0627 13:55:43.101515 (+ 20059us) write_op.cc:270] Start()
0627 13:55:43.101539 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540493687 usec, L: 0
0627 13:55:43.101655 (+ 116us) write_op.cc:270] Start()
0627 13:55:43.101672 (+ 17us) write_op.cc:276] Timestamp: P: 1751032540498317 usec, L: 0
0627 13:55:43.103929 (+ 2257us) write_op.cc:270] Start()
0627 13:55:43.103952 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540558985 usec, L: 0
0627 13:55:43.104094 (+ 142us) write_op.cc:270] Start()
0627 13:55:43.104139 (+ 45us) write_op.cc:276] Timestamp: P: 1751032540563952 usec, L: 0
0627 13:55:43.104275 (+ 136us) write_op.cc:270] Start()
0627 13:55:43.104293 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540564559 usec, L: 0
0627 13:55:43.104404 (+ 111us) write_op.cc:270] Start()
0627 13:55:43.104420 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540605754 usec, L: 0
0627 13:55:43.104533 (+ 113us) write_op.cc:270] Start()
0627 13:55:43.104549 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540614941 usec, L: 0
0627 13:55:43.104668 (+ 119us) write_op.cc:270] Start()
0627 13:55:43.104684 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540659293 usec, L: 0
0627 13:55:43.104830 (+ 146us) write_op.cc:270] Start()
0627 13:55:43.104849 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540660041 usec, L: 0
0627 13:55:43.104993 (+ 144us) write_op.cc:270] Start()
0627 13:55:43.105014 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540685395 usec, L: 0
0627 13:55:43.105203 (+ 189us) write_op.cc:270] Start()
0627 13:55:43.105233 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540693842 usec, L: 0
0627 13:55:43.105980 (+ 747us) write_op.cc:270] Start()
0627 13:55:43.106005 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540725116 usec, L: 0
0627 13:55:43.106196 (+ 191us) write_op.cc:270] Start()
0627 13:55:43.106215 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540733645 usec, L: 0
0627 13:55:43.106347 (+ 132us) write_op.cc:270] Start()
0627 13:55:43.106365 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540770855 usec, L: 0
0627 13:55:43.106498 (+ 133us) write_op.cc:270] Start()
0627 13:55:43.106517 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540785419 usec, L: 0
0627 13:55:43.106636 (+ 119us) write_op.cc:270] Start()
0627 13:55:43.106653 (+ 17us) write_op.cc:276] Timestamp: P: 1751032540812987 usec, L: 0
0627 13:55:43.106756 (+ 103us) write_op.cc:270] Start()
0627 13:55:43.106771 (+ 15us) write_op.cc:276] Timestamp: P: 1751032540815306 usec, L: 0
0627 13:55:43.107063 (+ 292us) write_op.cc:270] Start()
0627 13:55:43.107083 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540842462 usec, L: 0
0627 13:55:43.107211 (+ 128us) write_op.cc:270] Start()
0627 13:55:43.107226 (+ 15us) write_op.cc:276] Timestamp: P: 1751032540855528 usec, L: 0
0627 13:55:43.107364 (+ 138us) write_op.cc:270] Start()
0627 13:55:43.107380 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540863143 usec, L: 0
0627 13:55:43.123799 (+ 16419us) write_op.cc:270] Start()
0627 13:55:43.123828 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540878348 usec, L: 0
0627 13:55:43.148595 (+ 24767us) write_op.cc:270] Start()
0627 13:55:43.148626 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540894622 usec, L: 0
0627 13:55:43.184276 (+ 35650us) write_op.cc:270] Start()
0627 13:55:43.184305 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540937491 usec, L: 0
0627 13:55:43.185035 (+ 730us) write_op.cc:270] Start()
0627 13:55:43.185075 (+ 40us) write_op.cc:276] Timestamp: P: 1751032540939274 usec, L: 0
0627 13:55:43.185257 (+ 182us) write_op.cc:270] Start()
0627 13:55:43.185279 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540950176 usec, L: 0
0627 13:55:43.187426 (+ 2147us) write_op.cc:270] Start()
0627 13:55:43.187452 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540962211 usec, L: 0
0627 13:55:43.191291 (+ 3839us) write_op.cc:270] Start()
0627 13:55:43.191306 (+ 15us) write_op.cc:276] Timestamp: P: 1751032540994636 usec, L: 0
0627 13:55:43.224981 (+ 33675us) write_op.cc:270] Start()
0627 13:55:43.225010 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541006802 usec, L: 0
0627 13:55:43.228097 (+ 3087us) write_op.cc:270] Start()
0627 13:55:43.228123 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541010892 usec, L: 0
0627 13:55:43.228311 (+ 188us) write_op.cc:270] Start()
0627 13:55:43.228334 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541021742 usec, L: 0
0627 13:55:43.230145 (+ 1811us) write_op.cc:270] Start()
0627 13:55:43.230171 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541063430 usec, L: 0
0627 13:55:43.230358 (+ 187us) write_op.cc:270] Start()
0627 13:55:43.240439 (+ 10081us) write_op.cc:276] Timestamp: P: 1751032541067086 usec, L: 0
0627 13:55:43.255682 (+ 15243us) write_op.cc:270] Start()
0627 13:55:43.255701 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541088905 usec, L: 0
0627 13:55:43.260424 (+ 4723us) write_op.cc:270] Start()
0627 13:55:43.260447 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541113408 usec, L: 0
0627 13:55:43.276602 (+ 16155us) write_op.cc:270] Start()
0627 13:55:43.276641 (+ 39us) write_op.cc:276] Timestamp: P: 1751032541144875 usec, L: 0
0627 13:55:43.278077 (+ 1436us) write_op.cc:270] Start()
0627 13:55:43.278107 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541159966 usec, L: 0
0627 13:55:43.284044 (+ 5937us) write_op.cc:270] Start()
0627 13:55:43.284069 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541170965 usec, L: 0
0627 13:55:43.310862 (+ 26793us) write_op.cc:270] Start()
0627 13:55:43.310892 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541196101 usec, L: 0
0627 13:55:43.311028 (+ 136us) write_op.cc:270] Start()
0627 13:55:43.311042 (+ 14us) write_op.cc:276] Timestamp: P: 1751032541226429 usec, L: 0
0627 13:55:43.318147 (+ 7105us) write_op.cc:270] Start()
0627 13:55:43.318174 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541227310 usec, L: 0
0627 13:55:43.324212 (+ 6038us) write_op.cc:270] Start()
0627 13:55:43.324242 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541266204 usec, L: 0
0627 13:55:43.324432 (+ 190us) write_op.cc:270] Start()
0627 13:55:43.324459 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541282006 usec, L: 0
0627 13:55:43.324653 (+ 194us) write_op.cc:270] Start()
0627 13:55:43.324673 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541302300 usec, L: 0
0627 13:55:43.324809 (+ 136us) write_op.cc:270] Start()
0627 13:55:43.324828 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541317266 usec, L: 0
0627 13:55:43.372864 (+ 48036us) write_op.cc:270] Start()
0627 13:55:43.372892 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541323494 usec, L: 0
0627 13:55:43.374978 (+ 2086us) write_op.cc:270] Start()
0627 13:55:43.375010 (+ 32us) write_op.cc:276] Timestamp: P: 1751032541335095 usec, L: 0
0627 13:55:43.378662 (+ 3652us) write_op.cc:270] Start()
0627 13:55:43.378695 (+ 33us) write_op.cc:276] Timestamp: P: 1751032541364087 usec, L: 0
0627 13:55:43.381423 (+ 2728us) write_op.cc:270] Start()
0627 13:55:43.381452 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541408380 usec, L: 0
0627 13:55:43.385295 (+ 3843us) write_op.cc:270] Start()
0627 13:55:43.385334 (+ 39us) write_op.cc:276] Timestamp: P: 1751032541412946 usec, L: 0
0627 13:55:43.389363 (+ 4029us) write_op.cc:270] Start()
0627 13:55:43.389390 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541446220 usec, L: 0
0627 13:55:43.394989 (+ 5599us) write_op.cc:270] Start()
0627 13:55:43.395012 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541454365 usec, L: 0
0627 13:55:43.409508 (+ 14496us) write_op.cc:270] Start()
0627 13:55:43.409538 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541481550 usec, L: 0
0627 13:55:43.420375 (+ 10837us) write_op.cc:270] Start()
0627 13:55:43.420412 (+ 37us) write_op.cc:276] Timestamp: P: 1751032541491240 usec, L: 0
0627 13:55:43.428147 (+ 7735us) write_op.cc:270] Start()
0627 13:55:43.428178 (+ 31us) write_op.cc:276] Timestamp: P: 1751032541498916 usec, L: 0
0627 13:55:43.428379 (+ 201us) write_op.cc:270] Start()
0627 13:55:43.428406 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541515494 usec, L: 0
0627 13:55:43.428569 (+ 163us) write_op.cc:270] Start()
0627 13:55:43.428590 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541558560 usec, L: 0
0627 13:55:43.428747 (+ 157us) write_op.cc:270] Start()
0627 13:55:43.428769 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541580753 usec, L: 0
0627 13:55:43.429116 (+ 347us) write_op.cc:270] Start()
0627 13:55:43.429140 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541586129 usec, L: 0
0627 13:55:43.429315 (+ 175us) write_op.cc:270] Start()
0627 13:55:43.429349 (+ 34us) write_op.cc:276] Timestamp: P: 1751032541591080 usec, L: 0
0627 13:55:43.451161 (+ 21812us) write_op.cc:270] Start()
0627 13:55:43.451194 (+ 33us) write_op.cc:276] Timestamp: P: 1751032541608405 usec, L: 0
0627 13:55:43.453854 (+ 2660us) write_op.cc:270] Start()
0627 13:55:43.453878 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541646384 usec, L: 0
0627 13:55:43.455063 (+ 1185us) write_op.cc:270] Start()
0627 13:55:43.455085 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541655121 usec, L: 0
0627 13:55:43.458147 (+ 3062us) write_op.cc:270] Start()
0627 13:55:43.458175 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541677775 usec, L: 0
0627 13:55:43.458415 (+ 240us) write_op.cc:270] Start()
0627 13:55:43.458454 (+ 39us) write_op.cc:276] Timestamp: P: 1751032541710051 usec, L: 0
0627 13:55:43.458619 (+ 165us) write_op.cc:270] Start()
0627 13:55:43.458640 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541718849 usec, L: 0
0627 13:55:43.458778 (+ 138us) write_op.cc:270] Start()
0627 13:55:43.458798 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541733442 usec, L: 0
0627 13:55:43.458949 (+ 151us) write_op.cc:270] Start()
0627 13:55:43.458969 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541762787 usec, L: 0
0627 13:55:43.459102 (+ 133us) write_op.cc:270] Start()
0627 13:55:43.459121 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541797533 usec, L: 0
0627 13:55:43.459301 (+ 180us) write_op.cc:270] Start()
0627 13:55:43.459322 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541823531 usec, L: 0
0627 13:55:43.459751 (+ 429us) write_op.cc:270] Start()
0627 13:55:43.459774 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541835982 usec, L: 0
0627 13:55:43.459924 (+ 150us) write_op.cc:270] Start()
0627 13:55:43.459944 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541848336 usec, L: 0
0627 13:55:43.460087 (+ 143us) write_op.cc:270] Start()
0627 13:55:43.460107 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541885472 usec, L: 0
0627 13:55:43.460329 (+ 222us) write_op.cc:270] Start()
0627 13:55:43.460351 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541887955 usec, L: 0
0627 13:55:43.460514 (+ 163us) write_op.cc:270] Start()
0627 13:55:43.460536 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541888871 usec, L: 0
0627 13:55:43.461297 (+ 761us) write_op.cc:270] Start()
0627 13:55:43.462683 (+ 1386us) write_op.cc:276] Timestamp: P: 1751032541917275 usec, L: 0
0627 13:55:43.462865 (+ 182us) write_op.cc:270] Start()
0627 13:55:43.462889 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541946051 usec, L: 0
0627 13:55:43.463046 (+ 157us) write_op.cc:270] Start()
0627 13:55:43.463064 (+ 18us) write_op.cc:276] Timestamp: P: 1751032541948086 usec, L: 0
0627 13:55:43.463190 (+ 126us) write_op.cc:270] Start()
0627 13:55:43.463205 (+ 15us) write_op.cc:276] Timestamp: P: 1751032541963153 usec, L: 0
0627 13:55:43.463292 (+ 87us) write_op.cc:270] Start()
0627 13:55:43.463304 (+ 12us) write_op.cc:276] Timestamp: P: 1751032542024519 usec, L: 0
0627 13:55:43.492166 (+ 28862us) write_op.cc:270] Start()
0627 13:55:43.495432 (+ 3266us) write_op.cc:276] Timestamp: P: 1751032542074829 usec, L: 0
0627 13:55:43.497316 (+ 1884us) write_op.cc:270] Start()
0627 13:55:43.497349 (+ 33us) write_op.cc:276] Timestamp: P: 1751032542083180 usec, L: 0
0627 13:55:43.497546 (+ 197us) write_op.cc:270] Start()
0627 13:55:43.497569 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542103296 usec, L: 0
0627 13:55:43.497746 (+ 177us) write_op.cc:270] Start()
0627 13:55:43.497775 (+ 29us) write_op.cc:276] Timestamp: P: 1751032542128906 usec, L: 0
0627 13:55:43.497943 (+ 168us) write_op.cc:270] Start()
0627 13:55:43.497967 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542132085 usec, L: 0
0627 13:55:43.498191 (+ 224us) write_op.cc:270] Start()
0627 13:55:43.498220 (+ 29us) write_op.cc:276] Timestamp: P: 1751032542132575 usec, L: 0
0627 13:55:43.498420 (+ 200us) write_op.cc:270] Start()
0627 13:55:43.498444 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542168749 usec, L: 0
0627 13:55:43.498608 (+ 164us) write_op.cc:270] Start()
0627 13:55:43.498630 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542176784 usec, L: 0
0627 13:55:43.498791 (+ 161us) write_op.cc:270] Start()
0627 13:55:43.498816 (+ 25us) write_op.cc:276] Timestamp: P: 1751032542181111 usec, L: 0
0627 13:55:43.511577 (+ 12761us) write_op.cc:270] Start()
0627 13:55:43.511613 (+ 36us) write_op.cc:276] Timestamp: P: 1751032542208238 usec, L: 0
0627 13:55:43.518079 (+ 6466us) write_op.cc:270] Start()
0627 13:55:43.518108 (+ 29us) write_op.cc:276] Timestamp: P: 1751032542224235 usec, L: 0
0627 13:55:43.530404 (+ 12296us) write_op.cc:270] Start()
0627 13:55:43.530436 (+ 32us) write_op.cc:276] Timestamp: P: 1751032542245866 usec, L: 0
0627 13:55:43.531479 (+ 1043us) write_op.cc:270] Start()
0627 13:55:43.531506 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542252470 usec, L: 0
0627 13:55:43.536557 (+ 5051us) write_op.cc:270] Start()
0627 13:55:43.536582 (+ 25us) write_op.cc:276] Timestamp: P: 1751032542267136 usec, L: 0
0627 13:55:43.540972 (+ 4390us) write_op.cc:270] Start()
0627 13:55:43.540999 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542320311 usec, L: 0
0627 13:55:43.541169 (+ 170us) write_op.cc:270] Start()
0627 13:55:43.541188 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542340040 usec, L: 0
0627 13:55:43.567996 (+ 26808us) write_op.cc:270] Start()
0627 13:55:43.592968 (+ 24972us) write_op.cc:276] Timestamp: P: 1751032542347998 usec, L: 0
0627 13:55:43.595118 (+ 2150us) write_op.cc:270] Start()
0627 13:55:43.595149 (+ 31us) write_op.cc:276] Timestamp: P: 1751032542356882 usec, L: 0
0627 13:55:43.615295 (+ 20146us) write_op.cc:270] Start()
0627 13:55:43.615326 (+ 31us) write_op.cc:276] Timestamp: P: 1751032542405855 usec, L: 0
0627 13:55:43.630327 (+ 15001us) write_op.cc:270] Start()
0627 13:55:43.630359 (+ 32us) write_op.cc:276] Timestamp: P: 1751032542407318 usec, L: 0
0627 13:55:43.632473 (+ 2114us) write_op.cc:270] Start()
0627 13:55:43.632497 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542428720 usec, L: 0
0627 13:55:43.635795 (+ 3298us) write_op.cc:270] Start()
0627 13:55:43.635823 (+ 28us) write_op.cc:276] Timestamp: P: 1751032542436098 usec, L: 0
0627 13:55:43.636451 (+ 628us) write_op.cc:270] Start()
0627 13:55:43.636480 (+ 29us) write_op.cc:276] Timestamp: P: 1751032542443375 usec, L: 0
0627 13:55:43.639985 (+ 3505us) write_op.cc:270] Start()
0627 13:55:43.640012 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542447208 usec, L: 0
0627 13:55:43.682534 (+ 42522us) log.cc:844] Serialized 106396 byte log entry
0627 13:55:43.682688 (+ 154us) raft_consensus.cc:1643] Marking committed up to 1178
0627 13:55:43.830980 (+148292us) spinlock_profiling.cc:243] Waited 93.9 ms on lock 0x616000067cb8. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243eea03d0 00007f243ef595f7 00007f243ef55507 00007f2442cb56ed 00007f243d17f3b6 00007f243d17f0e5 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf
0627 13:55:43.830990 (+ 10us) raft_consensus.cc:1695] Filling consensus response to leader.
0627 13:55:43.831085 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0627 13:55:43.831098 (+ 13us) raft_consensus.cc:1683] finished
0627 13:55:43.831100 (+ 2us) raft_consensus.cc:1689] UpdateReplicas() finished
0627 13:55:43.831188 (+ 88us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0627 13:55:42.834734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.834849 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.834855 (+ 6us) write_op.cc:435] Acquired schema lock
0627 13:55:42.834858 (+ 3us) tablet.cc:660] Decoding operations
0627 13:55:42.835015 (+ 157us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.835029 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.835033 (+ 4us) tablet.cc:683] Acquiring locks for 6 operations
0627 13:55:42.835137 (+ 104us) tablet.cc:699] Row locks acquired
0627 13:55:42.835139 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.682735 (+847596us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.835352 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.835452 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.835459 (+ 7us) write_op.cc:435] Acquired schema lock
0627 13:55:42.835461 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.835603 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.835616 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.835621 (+ 5us) tablet.cc:683] Acquiring locks for 8 operations
0627 13:55:42.835705 (+ 84us) tablet.cc:699] Row locks acquired
0627 13:55:42.835707 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.682783 (+847076us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.835797 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.845518 (+ 9721us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.845524 (+ 6us) write_op.cc:435] Acquired schema lock
0627 13:55:42.845526 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.845645 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.845656 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.845662 (+ 6us) tablet.cc:683] Acquiring locks for 7 operations
0627 13:55:42.847549 (+ 1887us) tablet.cc:699] Row locks acquired
0627 13:55:42.847551 (+ 2us) write_op.cc:260] PREPARE: finished
0627 13:55:43.682818 (+835267us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.847611 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.847658 (+ 47us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.847661 (+ 3us) write_op.cc:435] Acquired schema lock
0627 13:55:42.847662 (+ 1us) tablet.cc:660] Decoding operations
0627 13:55:42.852159 (+ 4497us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.852165 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.852167 (+ 2us) tablet.cc:683] Acquiring locks for 7 operations
0627 13:55:42.855504 (+ 3337us) tablet.cc:699] Row locks acquired
0627 13:55:42.855507 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:42.857649 (+ 2142us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x61100008b3d8. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243a24c4cf 00007f24402f1684 00007f24402efebd 00007f24402ef379 00007f24402f77a0 00007f24402f75ad 0000000000a7573c 00007f24335589bf 00007f243355bc8b 00007f243355bb08
0627 13:55:43.682849 (+825200us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.857708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.857806 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0627 13:55:42.857811 (+ 5us) write_op.cc:435] Acquired schema lock
0627 13:55:42.857813 (+ 2us) tablet.cc:660] Decoding operations
0627 13:55:42.857907 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0627 13:55:42.857919 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0627 13:55:42.857924 (+ 5us) tablet.cc:683] Acquiring locks for 6 operations
0627 13:55:42.858006 (+ 82us) tablet.cc:699] Row locks acquired
0627 13:55:42.858009 (+ 3us) write_op.cc:260] PREPARE: finished
0627 13:55:43.682878 (+824869us) op_driver.cc:464] REPLICATION: finished
Related trace 'op':
0627 13:55:42.858119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet f55dc8052c494b8d8a79c2e7f9ed39d4
0627 13:55:42.858196 (+ 77us) write_op.cc:432] Acquiring schema lock in s
I20250627 13:55:44.225975 9931 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:44.226349 9931 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:44.228325 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:44.231766 9931 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:44.232009 9931 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:44.257076 9931 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:44.258788 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1181 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:44.268064 9931 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:44.286972 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1090 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:44.287048 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1181 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:44.307961 9929 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:44.309074 9929 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:44.310573 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1242 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:44.311765 9761 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:44.314116 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "82aebc7eff8c4c8a86eb994bf24bd99c" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1242 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:44.320688 9929 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:44.320870 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:44.321864 9932 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:44.325623 9761 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:44.329666 9931 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:44.330266 9929 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:43.209523 9829 rpcz_store.cc:276] 0627 13:55:40.899817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:40.899897 (+ 80us) service_pool.cc:225] Handling call
0627 13:55:41.013596 (+113699us) raft_consensus.cc:1469] Updating replica for 331 ops
0627 13:55:41.155720 (+142124us) raft_consensus.cc:1532] Early marking committed up to index 759
0627 13:55:41.156100 (+ 380us) raft_consensus.cc:1537] Triggering prepare for 331 ops
0627 13:55:41.156219 (+ 119us) write_op.cc:270] Start()
0627 13:55:41.156242 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534282918 usec, L: 0
0627 13:55:41.177449 (+ 21207us) write_op.cc:270] Start()
0627 13:55:41.177478 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534349245 usec, L: 0
0627 13:55:41.177715 (+ 237us) write_op.cc:270] Start()
0627 13:55:41.177740 (+ 25us) write_op.cc:276] Timestamp: P: 1751032534350710 usec, L: 0
0627 13:55:41.177924 (+ 184us) write_op.cc:270] Start()
0627 13:55:41.177947 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534363263 usec, L: 0
0627 13:55:41.178152 (+ 205us) write_op.cc:270] Start()
0627 13:55:41.178174 (+ 22us) write_op.cc:276] Timestamp: P: 1751032534382835 usec, L: 0
0627 13:55:41.178364 (+ 190us) write_op.cc:270] Start()
0627 13:55:41.178391 (+ 27us) write_op.cc:276] Timestamp: P: 1751032534429471 usec, L: 0
0627 13:55:41.178570 (+ 179us) write_op.cc:270] Start()
0627 13:55:41.183798 (+ 5228us) write_op.cc:276] Timestamp: P: 1751032534439265 usec, L: 0
0627 13:55:41.184054 (+ 256us) write_op.cc:270] Start()
0627 13:55:41.184082 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534440130 usec, L: 0
0627 13:55:41.184267 (+ 185us) write_op.cc:270] Start()
0627 13:55:41.184287 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534444067 usec, L: 0
0627 13:55:41.184558 (+ 271us) write_op.cc:270] Start()
0627 13:55:41.184603 (+ 45us) write_op.cc:276] Timestamp: P: 1751032534478359 usec, L: 0
0627 13:55:41.184760 (+ 157us) write_op.cc:270] Start()
0627 13:55:41.184781 (+ 21us) write_op.cc:276] Timestamp: P: 1751032534513531 usec, L: 0
0627 13:55:41.184953 (+ 172us) write_op.cc:270] Start()
0627 13:55:41.184976 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534519570 usec, L: 0
0627 13:55:41.185129 (+ 153us) write_op.cc:270] Start()
0627 13:55:41.185149 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534544421 usec, L: 0
0627 13:55:41.185286 (+ 137us) write_op.cc:270] Start()
0627 13:55:41.185307 (+ 21us) write_op.cc:276] Timestamp: P: 1751032534568960 usec, L: 0
0627 13:55:41.185495 (+ 188us) write_op.cc:270] Start()
0627 13:55:41.185518 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534598915 usec, L: 0
0627 13:55:41.185713 (+ 195us) write_op.cc:270] Start()
0627 13:55:41.185735 (+ 22us) write_op.cc:276] Timestamp: P: 1751032534618599 usec, L: 0
0627 13:55:41.185921 (+ 186us) write_op.cc:270] Start()
0627 13:55:41.185943 (+ 22us) write_op.cc:276] Timestamp: P: 1751032534625404 usec, L: 0
0627 13:55:41.186125 (+ 182us) write_op.cc:270] Start()
0627 13:55:41.186146 (+ 21us) write_op.cc:276] Timestamp: P: 1751032534689227 usec, L: 0
0627 13:55:41.186318 (+ 172us) write_op.cc:270] Start()
0627 13:55:41.186338 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534691909 usec, L: 0
0627 13:55:41.186515 (+ 177us) write_op.cc:270] Start()
0627 13:55:41.186535 (+ 20us) write_op.cc:276] Timestamp: P: 1751032534709439 usec, L: 0
0627 13:55:41.186686 (+ 151us) write_op.cc:270] Start()
0627 13:55:41.186712 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534737455 usec, L: 0
0627 13:55:41.186850 (+ 138us) write_op.cc:270] Start()
0627 13:55:41.186883 (+ 33us) write_op.cc:276] Timestamp: P: 1751032534756141 usec, L: 0
0627 13:55:41.196045 (+ 9162us) write_op.cc:270] Start()
0627 13:55:41.196071 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534801111 usec, L: 0
0627 13:55:41.198073 (+ 2002us) write_op.cc:270] Start()
0627 13:55:41.198101 (+ 28us) write_op.cc:276] Timestamp: P: 1751032534851251 usec, L: 0
0627 13:55:41.204553 (+ 6452us) write_op.cc:270] Start()
0627 13:55:41.204579 (+ 26us) write_op.cc:276] Timestamp: P: 1751032534852056 usec, L: 0
0627 13:55:41.222355 (+ 17776us) write_op.cc:270] Start()
0627 13:55:41.222385 (+ 30us) write_op.cc:276] Timestamp: P: 1751032534863381 usec, L: 0
0627 13:55:41.234200 (+ 11815us) write_op.cc:270] Start()
0627 13:55:41.234224 (+ 24us) write_op.cc:276] Timestamp: P: 1751032534892957 usec, L: 0
0627 13:55:41.270612 (+ 36388us) write_op.cc:270] Start()
0627 13:55:41.270635 (+ 23us) write_op.cc:276] Timestamp: P: 1751032534958822 usec, L: 0
0627 13:55:41.283956 (+ 13321us) write_op.cc:270] Start()
0627 13:55:41.283985 (+ 29us) write_op.cc:276] Timestamp: P: 1751032534990483 usec, L: 0
0627 13:55:41.286720 (+ 2735us) write_op.cc:270] Start()
0627 13:55:41.286752 (+ 32us) write_op.cc:276] Timestamp: P: 1751032534991170 usec, L: 0
0627 13:55:41.292732 (+ 5980us) write_op.cc:270] Start()
0627 13:55:41.292762 (+ 30us) write_op.cc:276] Timestamp: P: 1751032534995016 usec, L: 0
0627 13:55:41.305234 (+ 12472us) write_op.cc:270] Start()
0627 13:55:41.305271 (+ 37us) write_op.cc:276] Timestamp: P: 1751032535058059 usec, L: 0
0627 13:55:41.305511 (+ 240us) write_op.cc:270] Start()
0627 13:55:41.305538 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535083942 usec, L: 0
0627 13:55:41.313688 (+ 8150us) write_op.cc:270] Start()
0627 13:55:41.313719 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535087867 usec, L: 0
0627 13:55:41.313931 (+ 212us) write_op.cc:270] Start()
0627 13:55:41.313955 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535109867 usec, L: 0
0627 13:55:41.334785 (+ 20830us) write_op.cc:270] Start()
0627 13:55:41.334819 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535112118 usec, L: 0
0627 13:55:41.335031 (+ 212us) write_op.cc:270] Start()
0627 13:55:41.335054 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535177025 usec, L: 0
0627 13:55:41.335196 (+ 142us) write_op.cc:270] Start()
0627 13:55:41.335214 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535179319 usec, L: 0
0627 13:55:41.335368 (+ 154us) write_op.cc:270] Start()
0627 13:55:41.335390 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535192199 usec, L: 0
0627 13:55:41.335564 (+ 174us) write_op.cc:270] Start()
0627 13:55:41.335586 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535193562 usec, L: 0
0627 13:55:41.364744 (+ 29158us) write_op.cc:270] Start()
0627 13:55:41.364775 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535227204 usec, L: 0
0627 13:55:41.373879 (+ 9104us) write_op.cc:270] Start()
0627 13:55:41.373917 (+ 38us) write_op.cc:276] Timestamp: P: 1751032535242871 usec, L: 0
0627 13:55:41.374141 (+ 224us) write_op.cc:270] Start()
0627 13:55:41.374165 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535249822 usec, L: 0
0627 13:55:41.379825 (+ 5660us) write_op.cc:270] Start()
0627 13:55:41.379853 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535257298 usec, L: 0
0627 13:55:41.387558 (+ 7705us) write_op.cc:270] Start()
0627 13:55:41.387580 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535290393 usec, L: 0
0627 13:55:41.387735 (+ 155us) write_op.cc:270] Start()
0627 13:55:41.387755 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535322268 usec, L: 0
0627 13:55:41.396160 (+ 8405us) write_op.cc:270] Start()
0627 13:55:41.396195 (+ 35us) write_op.cc:276] Timestamp: P: 1751032535323706 usec, L: 0
0627 13:55:41.396399 (+ 204us) write_op.cc:270] Start()
0627 13:55:41.396422 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535332515 usec, L: 0
0627 13:55:41.396570 (+ 148us) write_op.cc:270] Start()
0627 13:55:41.396589 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535353085 usec, L: 0
0627 13:55:41.396736 (+ 147us) write_op.cc:270] Start()
0627 13:55:41.396757 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535361497 usec, L: 0
0627 13:55:41.396910 (+ 153us) write_op.cc:270] Start()
0627 13:55:41.396943 (+ 33us) write_op.cc:276] Timestamp: P: 1751032535372820 usec, L: 0
0627 13:55:41.397073 (+ 130us) write_op.cc:270] Start()
0627 13:55:41.397097 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535404044 usec, L: 0
0627 13:55:41.397412 (+ 315us) write_op.cc:270] Start()
0627 13:55:41.397436 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535411003 usec, L: 0
0627 13:55:41.398494 (+ 1058us) write_op.cc:270] Start()
0627 13:55:41.398523 (+ 29us) write_op.cc:276] Timestamp: P: 1751032535433516 usec, L: 0
0627 13:55:41.398705 (+ 182us) write_op.cc:270] Start()
0627 13:55:41.398724 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535435090 usec, L: 0
0627 13:55:41.398855 (+ 131us) write_op.cc:270] Start()
0627 13:55:41.398869 (+ 14us) write_op.cc:276] Timestamp: P: 1751032535474912 usec, L: 0
0627 13:55:41.398959 (+ 90us) write_op.cc:270] Start()
0627 13:55:41.398971 (+ 12us) write_op.cc:276] Timestamp: P: 1751032535485888 usec, L: 0
0627 13:55:41.399081 (+ 110us) write_op.cc:270] Start()
0627 13:55:41.399099 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535497612 usec, L: 0
0627 13:55:41.399250 (+ 151us) write_op.cc:270] Start()
0627 13:55:41.399268 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535504979 usec, L: 0
0627 13:55:41.399419 (+ 151us) write_op.cc:270] Start()
0627 13:55:41.399457 (+ 38us) write_op.cc:276] Timestamp: P: 1751032535519346 usec, L: 0
0627 13:55:41.399872 (+ 415us) write_op.cc:270] Start()
0627 13:55:41.399897 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535582004 usec, L: 0
0627 13:55:41.400035 (+ 138us) write_op.cc:270] Start()
0627 13:55:41.400051 (+ 16us) write_op.cc:276] Timestamp: P: 1751032535598224 usec, L: 0
0627 13:55:41.400191 (+ 140us) write_op.cc:270] Start()
0627 13:55:41.400212 (+ 21us) write_op.cc:276] Timestamp: P: 1751032535602317 usec, L: 0
0627 13:55:41.400360 (+ 148us) write_op.cc:270] Start()
0627 13:55:41.400380 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535619369 usec, L: 0
0627 13:55:41.400544 (+ 164us) write_op.cc:270] Start()
0627 13:55:41.400563 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535663319 usec, L: 0
0627 13:55:41.400718 (+ 155us) write_op.cc:270] Start()
0627 13:55:41.400738 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535666487 usec, L: 0
0627 13:55:41.400911 (+ 173us) write_op.cc:270] Start()
0627 13:55:41.400934 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535675574 usec, L: 0
0627 13:55:41.401086 (+ 152us) write_op.cc:270] Start()
0627 13:55:41.401108 (+ 22us) write_op.cc:276] Timestamp: P: 1751032535732846 usec, L: 0
0627 13:55:41.401272 (+ 164us) write_op.cc:270] Start()
0627 13:55:41.401296 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535739730 usec, L: 0
0627 13:55:41.401457 (+ 161us) write_op.cc:270] Start()
0627 13:55:41.401477 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535740398 usec, L: 0
0627 13:55:41.401634 (+ 157us) write_op.cc:270] Start()
0627 13:55:41.401653 (+ 19us) write_op.cc:276] Timestamp: P: 1751032535745923 usec, L: 0
0627 13:55:41.401818 (+ 165us) write_op.cc:270] Start()
0627 13:55:41.401835 (+ 17us) write_op.cc:276] Timestamp: P: 1751032535790213 usec, L: 0
0627 13:55:41.401966 (+ 131us) write_op.cc:270] Start()
0627 13:55:41.401984 (+ 18us) write_op.cc:276] Timestamp: P: 1751032535792791 usec, L: 0
0627 13:55:41.448020 (+ 46036us) write_op.cc:270] Start()
0627 13:55:41.448051 (+ 31us) write_op.cc:276] Timestamp: P: 1751032535798238 usec, L: 0
0627 13:55:41.454570 (+ 6519us) write_op.cc:270] Start()
0627 13:55:41.454606 (+ 36us) write_op.cc:276] Timestamp: P: 1751032535814130 usec, L: 0
0627 13:55:41.456233 (+ 1627us) write_op.cc:270] Start()
0627 13:55:41.456261 (+ 28us) write_op.cc:276] Timestamp: P: 1751032535845780 usec, L: 0
0627 13:55:41.458301 (+ 2040us) write_op.cc:270] Start()
0627 13:55:41.458324 (+ 23us) write_op.cc:276] Timestamp: P: 1751032535851864 usec, L: 0
0627 13:55:41.458494 (+ 170us) write_op.cc:270] Start()
0627 13:55:41.458518 (+ 24us) write_op.cc:276] Timestamp: P: 1751032535856113 usec, L: 0
0627 13:55:41.475525 (+ 17007us) write_op.cc:270] Start()
0627 13:55:41.475559 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535896282 usec, L: 0
0627 13:55:41.475784 (+ 225us) write_op.cc:270] Start()
0627 13:55:41.475809 (+ 25us) write_op.cc:276] Timestamp: P: 1751032535902061 usec, L: 0
0627 13:55:41.476000 (+ 191us) write_op.cc:270] Start()
0627 13:55:41.476020 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535920591 usec, L: 0
0627 13:55:41.476193 (+ 173us) write_op.cc:270] Start()
0627 13:55:41.476213 (+ 20us) write_op.cc:276] Timestamp: P: 1751032535938302 usec, L: 0
0627 13:55:41.485978 (+ 9765us) write_op.cc:270] Start()
0627 13:55:41.486012 (+ 34us) write_op.cc:276] Timestamp: P: 1751032535960147 usec, L: 0
0627 13:55:41.486263 (+ 251us) write_op.cc:270] Start()
0627 13:55:41.486290 (+ 27us) write_op.cc:276] Timestamp: P: 1751032535995635 usec, L: 0
0627 13:55:41.506089 (+ 19799us) write_op.cc:270] Start()
0627 13:55:41.506124 (+ 35us) write_op.cc:276] Timestamp: P: 1751032536011257 usec, L: 0
0627 13:55:41.515666 (+ 9542us) write_op.cc:270] Start()
0627 13:55:41.515690 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536041551 usec, L: 0
0627 13:55:41.517895 (+ 2205us) write_op.cc:270] Start()
0627 13:55:41.517931 (+ 36us) write_op.cc:276] Timestamp: P: 1751032536077415 usec, L: 0
0627 13:55:41.518175 (+ 244us) write_op.cc:270] Start()
0627 13:55:41.518202 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536093127 usec, L: 0
0627 13:55:41.592228 (+ 74026us) write_op.cc:270] Start()
0627 13:55:41.592259 (+ 31us) write_op.cc:276] Timestamp: P: 1751032536144616 usec, L: 0
0627 13:55:41.592486 (+ 227us) write_op.cc:270] Start()
0627 13:55:41.592510 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536154350 usec, L: 0
0627 13:55:41.592670 (+ 160us) write_op.cc:270] Start()
0627 13:55:41.592692 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536197406 usec, L: 0
0627 13:55:41.592848 (+ 156us) write_op.cc:270] Start()
0627 13:55:41.592868 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536208358 usec, L: 0
0627 13:55:41.593067 (+ 199us) write_op.cc:270] Start()
0627 13:55:41.593090 (+ 23us) write_op.cc:276] Timestamp: P: 1751032536229974 usec, L: 0
0627 13:55:41.593250 (+ 160us) write_op.cc:270] Start()
0627 13:55:41.593270 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536248999 usec, L: 0
0627 13:55:41.593415 (+ 145us) write_op.cc:270] Start()
0627 13:55:41.593443 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536253967 usec, L: 0
0627 13:55:41.593596 (+ 153us) write_op.cc:270] Start()
0627 13:55:41.593616 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536261817 usec, L: 0
0627 13:55:41.593777 (+ 161us) write_op.cc:270] Start()
0627 13:55:41.593797 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536326003 usec, L: 0
0627 13:55:41.593939 (+ 142us) write_op.cc:270] Start()
0627 13:55:41.593968 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536327997 usec, L: 0
0627 13:55:41.594184 (+ 216us) write_op.cc:270] Start()
0627 13:55:41.594213 (+ 29us) write_op.cc:276] Timestamp: P: 1751032536334995 usec, L: 0
0627 13:55:41.594361 (+ 148us) write_op.cc:270] Start()
0627 13:55:41.594380 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536338862 usec, L: 0
0627 13:55:41.594551 (+ 171us) write_op.cc:270] Start()
0627 13:55:41.594572 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536394014 usec, L: 0
0627 13:55:41.594714 (+ 142us) write_op.cc:270] Start()
0627 13:55:41.594734 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536395367 usec, L: 0
0627 13:55:41.594907 (+ 173us) write_op.cc:270] Start()
0627 13:55:41.594927 (+ 20us) write_op.cc:276] Timestamp: P: 1751032536397742 usec, L: 0
0627 13:55:41.614970 (+ 20043us) write_op.cc:270] Start()
0627 13:55:41.615017 (+ 47us) write_op.cc:276] Timestamp: P: 1751032536414474 usec, L: 0
0627 13:55:41.615892 (+ 875us) write_op.cc:270] Start()
0627 13:55:41.615918 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536458416 usec, L: 0
0627 13:55:41.617478 (+ 1560us) write_op.cc:270] Start()
0627 13:55:41.617504 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536460715 usec, L: 0
0627 13:55:41.617662 (+ 158us) write_op.cc:270] Start()
0627 13:55:41.617680 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536501460 usec, L: 0
0627 13:55:41.617809 (+ 129us) write_op.cc:270] Start()
0627 13:55:41.617828 (+ 19us) write_op.cc:276] Timestamp: P: 1751032536508269 usec, L: 0
0627 13:55:41.619400 (+ 1572us) write_op.cc:270] Start()
0627 13:55:41.619424 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536525095 usec, L: 0
0627 13:55:41.619591 (+ 167us) write_op.cc:270] Start()
0627 13:55:41.619613 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536530235 usec, L: 0
0627 13:55:41.619803 (+ 190us) write_op.cc:270] Start()
0627 13:55:41.621846 (+ 2043us) write_op.cc:276] Timestamp: P: 1751032536579427 usec, L: 0
0627 13:55:41.624099 (+ 2253us) write_op.cc:270] Start()
0627 13:55:41.624129 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536587035 usec, L: 0
0627 13:55:41.624834 (+ 705us) write_op.cc:270] Start()
0627 13:55:41.624856 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536607641 usec, L: 0
0627 13:55:41.625086 (+ 230us) write_op.cc:270] Start()
0627 13:55:41.625107 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536608355 usec, L: 0
0627 13:55:41.625248 (+ 141us) write_op.cc:270] Start()
0627 13:55:41.625266 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536638673 usec, L: 0
0627 13:55:41.625395 (+ 129us) write_op.cc:270] Start()
0627 13:55:41.625413 (+ 18us) write_op.cc:276] Timestamp: P: 1751032536645290 usec, L: 0
0627 13:55:41.625784 (+ 371us) write_op.cc:270] Start()
0627 13:55:41.625805 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536657895 usec, L: 0
0627 13:55:41.647525 (+ 21720us) write_op.cc:270] Start()
0627 13:55:41.647553 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536677883 usec, L: 0
0627 13:55:41.649071 (+ 1518us) write_op.cc:270] Start()
0627 13:55:41.649101 (+ 30us) write_op.cc:276] Timestamp: P: 1751032536706178 usec, L: 0
0627 13:55:41.660517 (+ 11416us) write_op.cc:270] Start()
0627 13:55:41.660589 (+ 72us) write_op.cc:276] Timestamp: P: 1751032536723452 usec, L: 0
0627 13:55:41.660956 (+ 367us) write_op.cc:270] Start()
0627 13:55:41.660983 (+ 27us) write_op.cc:276] Timestamp: P: 1751032536741369 usec, L: 0
0627 13:55:41.679506 (+ 18523us) write_op.cc:270] Start()
0627 13:55:41.679527 (+ 21us) write_op.cc:276] Timestamp: P: 1751032536759952 usec, L: 0
0627 13:55:41.685382 (+ 5855us) write_op.cc:270] Start()
0627 13:55:41.685410 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536785287 usec, L: 0
0627 13:55:41.685572 (+ 162us) write_op.cc:270] Start()
0627 13:55:41.685594 (+ 22us) write_op.cc:276] Timestamp: P: 1751032536799118 usec, L: 0
0627 13:55:41.700172 (+ 14578us) write_op.cc:270] Start()
0627 13:55:41.700204 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536825723 usec, L: 0
0627 13:55:41.706504 (+ 6300us) write_op.cc:270] Start()
0627 13:55:41.706530 (+ 26us) write_op.cc:276] Timestamp: P: 1751032536831763 usec, L: 0
0627 13:55:41.755449 (+ 48919us) write_op.cc:270] Start()
0627 13:55:41.755495 (+ 46us) write_op.cc:276] Timestamp: P: 1751032536849467 usec, L: 0
0627 13:55:41.763005 (+ 7510us) write_op.cc:270] Start()
0627 13:55:41.763037 (+ 32us) write_op.cc:276] Timestamp: P: 1751032536876523 usec, L: 0
0627 13:55:41.774371 (+ 11334us) write_op.cc:270] Start()
0627 13:55:41.774399 (+ 28us) write_op.cc:276] Timestamp: P: 1751032536898174 usec, L: 0
0627 13:55:41.849653 (+ 75254us) write_op.cc:270] Start()
0627 13:55:41.849684 (+ 31us) write_op.cc:276] Timestamp: P: 1751032536903483 usec, L: 0
0627 13:55:41.852049 (+ 2365us) write_op.cc:270] Start()
0627 13:55:41.852116 (+ 67us) write_op.cc:276] Timestamp: P: 1751032536910843 usec, L: 0
0627 13:55:41.852309 (+ 193us) write_op.cc:270] Start()
0627 13:55:41.852334 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536972360 usec, L: 0
0627 13:55:41.852775 (+ 441us) write_op.cc:270] Start()
0627 13:55:41.852800 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536982843 usec, L: 0
0627 13:55:41.854533 (+ 1733us) write_op.cc:270] Start()
0627 13:55:41.854557 (+ 24us) write_op.cc:276] Timestamp: P: 1751032536989688 usec, L: 0
0627 13:55:41.854735 (+ 178us) write_op.cc:270] Start()
0627 13:55:41.854760 (+ 25us) write_op.cc:276] Timestamp: P: 1751032536992829 usec, L: 0
0627 13:55:41.854930 (+ 170us) write_op.cc:270] Start()
0627 13:55:41.854951 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537037872 usec, L: 0
0627 13:55:41.855115 (+ 164us) write_op.cc:270] Start()
0627 13:55:41.855138 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537050446 usec, L: 0
0627 13:55:41.858110 (+ 2972us) write_op.cc:270] Start()
0627 13:55:41.858136 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537062335 usec, L: 0
0627 13:55:41.858321 (+ 185us) write_op.cc:270] Start()
0627 13:55:41.858343 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537086223 usec, L: 0
0627 13:55:41.858504 (+ 161us) write_op.cc:270] Start()
0627 13:55:41.858524 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537092340 usec, L: 0
0627 13:55:41.858655 (+ 131us) write_op.cc:270] Start()
0627 13:55:41.858675 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537167267 usec, L: 0
0627 13:55:41.858810 (+ 135us) write_op.cc:270] Start()
0627 13:55:41.858833 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537204951 usec, L: 0
0627 13:55:41.858955 (+ 122us) write_op.cc:270] Start()
0627 13:55:41.858977 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537236289 usec, L: 0
0627 13:55:41.859081 (+ 104us) write_op.cc:270] Start()
0627 13:55:41.859093 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537245457 usec, L: 0
0627 13:55:41.859177 (+ 84us) write_op.cc:270] Start()
0627 13:55:41.859189 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537255387 usec, L: 0
0627 13:55:41.859284 (+ 95us) write_op.cc:270] Start()
0627 13:55:41.859295 (+ 11us) write_op.cc:276] Timestamp: P: 1751032537283643 usec, L: 0
0627 13:55:41.859379 (+ 84us) write_op.cc:270] Start()
0627 13:55:41.859391 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537296463 usec, L: 0
0627 13:55:41.859471 (+ 80us) write_op.cc:270] Start()
0627 13:55:41.859487 (+ 16us) write_op.cc:276] Timestamp: P: 1751032537323814 usec, L: 0
0627 13:55:41.859635 (+ 148us) write_op.cc:270] Start()
0627 13:55:41.859656 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537331336 usec, L: 0
0627 13:55:41.859811 (+ 155us) write_op.cc:270] Start()
0627 13:55:41.859830 (+ 19us) write_op.cc:276] Timestamp: P: 1751032537357558 usec, L: 0
0627 13:55:41.859994 (+ 164us) write_op.cc:270] Start()
0627 13:55:41.860020 (+ 26us) write_op.cc:276] Timestamp: P: 1751032537372161 usec, L: 0
0627 13:55:41.860170 (+ 150us) write_op.cc:270] Start()
0627 13:55:41.860190 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537382001 usec, L: 0
0627 13:55:41.860322 (+ 132us) write_op.cc:270] Start()
0627 13:55:41.860341 (+ 19us) write_op.cc:276] Timestamp: P: 1751032537426338 usec, L: 0
0627 13:55:41.860498 (+ 157us) write_op.cc:270] Start()
0627 13:55:41.860519 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537433572 usec, L: 0
0627 13:55:41.860704 (+ 185us) write_op.cc:270] Start()
0627 13:55:41.860726 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537462239 usec, L: 0
0627 13:55:41.860876 (+ 150us) write_op.cc:270] Start()
0627 13:55:41.860897 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537473118 usec, L: 0
0627 13:55:41.861069 (+ 172us) write_op.cc:270] Start()
0627 13:55:41.861092 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537473711 usec, L: 0
0627 13:55:41.861259 (+ 167us) write_op.cc:270] Start()
0627 13:55:41.861283 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537492702 usec, L: 0
0627 13:55:41.861448 (+ 165us) write_op.cc:270] Start()
0627 13:55:41.861470 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537516018 usec, L: 0
0627 13:55:41.861614 (+ 144us) write_op.cc:270] Start()
0627 13:55:41.861637 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537541177 usec, L: 0
0627 13:55:41.861782 (+ 145us) write_op.cc:270] Start()
0627 13:55:41.861803 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537544546 usec, L: 0
0627 13:55:41.861964 (+ 161us) write_op.cc:270] Start()
0627 13:55:41.861998 (+ 34us) write_op.cc:276] Timestamp: P: 1751032537555188 usec, L: 0
0627 13:55:41.877481 (+ 15483us) write_op.cc:270] Start()
0627 13:55:41.877501 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537588133 usec, L: 0
0627 13:55:41.894815 (+ 17314us) write_op.cc:270] Start()
0627 13:55:41.894842 (+ 27us) write_op.cc:276] Timestamp: P: 1751032537605100 usec, L: 0
0627 13:55:41.894995 (+ 153us) write_op.cc:270] Start()
0627 13:55:41.895011 (+ 16us) write_op.cc:276] Timestamp: P: 1751032537611621 usec, L: 0
0627 13:55:41.895100 (+ 89us) write_op.cc:270] Start()
0627 13:55:41.895112 (+ 12us) write_op.cc:276] Timestamp: P: 1751032537623112 usec, L: 0
0627 13:55:41.916271 (+ 21159us) write_op.cc:270] Start()
0627 13:55:41.916317 (+ 46us) write_op.cc:276] Timestamp: P: 1751032537688832 usec, L: 0
0627 13:55:41.916561 (+ 244us) write_op.cc:270] Start()
0627 13:55:41.916586 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537689422 usec, L: 0
0627 13:55:41.916780 (+ 194us) write_op.cc:270] Start()
0627 13:55:41.916809 (+ 29us) write_op.cc:276] Timestamp: P: 1751032537699711 usec, L: 0
0627 13:55:41.916998 (+ 189us) write_op.cc:270] Start()
0627 13:55:41.917023 (+ 25us) write_op.cc:276] Timestamp: P: 1751032537726280 usec, L: 0
0627 13:55:41.917177 (+ 154us) write_op.cc:270] Start()
0627 13:55:41.917196 (+ 19us) write_op.cc:276] Timestamp: P: 1751032537745371 usec, L: 0
0627 13:55:41.917336 (+ 140us) write_op.cc:270] Start()
0627 13:55:41.917357 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537776961 usec, L: 0
0627 13:55:41.917495 (+ 138us) write_op.cc:270] Start()
0627 13:55:41.917512 (+ 17us) write_op.cc:276] Timestamp: P: 1751032537781214 usec, L: 0
0627 13:55:41.917646 (+ 134us) write_op.cc:270] Start()
0627 13:55:41.917665 (+ 19us) write_op.cc:276] Timestamp: P: 1751032537783877 usec, L: 0
0627 13:55:41.917838 (+ 173us) write_op.cc:270] Start()
0627 13:55:41.917861 (+ 23us) write_op.cc:276] Timestamp: P: 1751032537821994 usec, L: 0
0627 13:55:41.918002 (+ 141us) write_op.cc:270] Start()
0627 13:55:41.918022 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537836421 usec, L: 0
0627 13:55:41.932784 (+ 14762us) write_op.cc:270] Start()
0627 13:55:41.932812 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537850029 usec, L: 0
0627 13:55:41.948760 (+ 15948us) write_op.cc:270] Start()
0627 13:55:41.948794 (+ 34us) write_op.cc:276] Timestamp: P: 1751032537865814 usec, L: 0
0627 13:55:41.948976 (+ 182us) write_op.cc:270] Start()
0627 13:55:41.949000 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537871978 usec, L: 0
0627 13:55:41.955962 (+ 6962us) write_op.cc:270] Start()
0627 13:55:41.955993 (+ 31us) write_op.cc:276] Timestamp: P: 1751032537904971 usec, L: 0
0627 13:55:41.974849 (+ 18856us) write_op.cc:270] Start()
0627 13:55:41.974883 (+ 34us) write_op.cc:276] Timestamp: P: 1751032537906992 usec, L: 0
0627 13:55:41.975110 (+ 227us) write_op.cc:270] Start()
0627 13:55:41.975138 (+ 28us) write_op.cc:276] Timestamp: P: 1751032537924010 usec, L: 0
0627 13:55:41.975326 (+ 188us) write_op.cc:270] Start()
0627 13:55:41.975350 (+ 24us) write_op.cc:276] Timestamp: P: 1751032537926734 usec, L: 0
0627 13:55:41.975515 (+ 165us) write_op.cc:270] Start()
0627 13:55:41.975537 (+ 22us) write_op.cc:276] Timestamp: P: 1751032537968763 usec, L: 0
0627 13:55:41.975676 (+ 139us) write_op.cc:270] Start()
0627 13:55:41.975696 (+ 20us) write_op.cc:276] Timestamp: P: 1751032537971800 usec, L: 0
0627 13:55:41.975873 (+ 177us) write_op.cc:270] Start()
0627 13:55:41.975894 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537972532 usec, L: 0
0627 13:55:41.976060 (+ 166us) write_op.cc:270] Start()
0627 13:55:41.976081 (+ 21us) write_op.cc:276] Timestamp: P: 1751032537986085 usec, L: 0
0627 13:55:41.976252 (+ 171us) write_op.cc:270] Start()
0627 13:55:41.976278 (+ 26us) write_op.cc:276] Timestamp: P: 1751032538037203 usec, L: 0
0627 13:55:41.976446 (+ 168us) write_op.cc:270] Start()
0627 13:55:41.976467 (+ 21us) write_op.cc:276] Timestamp: P: 1751032538051651 usec, L: 0
0627 13:55:41.976601 (+ 134us) write_op.cc:270] Start()
0627 13:55:41.976623 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538077298 usec, L: 0
0627 13:55:42.001457 (+ 24834us) write_op.cc:270] Start()
0627 13:55:42.001482 (+ 25us) write_op.cc:276] Timestamp: P: 1751032538083384 usec, L: 0
0627 13:55:42.065417 (+ 63935us) write_op.cc:270] Start()
0627 13:55:42.065438 (+ 21us) write_op.cc:276] Timestamp: P: 1751032538100765 usec, L: 0
0627 13:55:42.065594 (+ 156us) write_op.cc:270] Start()
0627 13:55:42.065615 (+ 21us) write_op.cc:276] Timestamp: P: 1751032538144673 usec, L: 0
0627 13:55:42.069661 (+ 4046us) write_op.cc:270] Start()
0627 13:55:42.069679 (+ 18us) write_op.cc:276] Timestamp: P: 1751032538163931 usec, L: 0
0627 13:55:42.090292 (+ 20613us) write_op.cc:270] Start()
0627 13:55:42.090319 (+ 27us) write_op.cc:276] Timestamp: P: 1751032538170624 usec, L: 0
0627 13:55:42.090498 (+ 179us) write_op.cc:270] Start()
0627 13:55:42.090520 (+ 22us) write_op.cc:276] Timestamp: P: 1751032538174892 usec, L: 0
0627 13:55:42.092516 (+ 1996us) write_op.cc:270] Start()
0627 13:55:42.092542 (+ 26us) write_op.cc:276] Timestamp: P: 175103
I20250627 13:55:44.926278 9932 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:44.926543 9932 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:44.928182 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1267 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:44.928905 9932 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:44.929694 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f55dc8052c494b8d8a79c2e7f9ed39d4" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1267 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:44.930434 9760 leader_election.cc:304] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:44.936841 9932 raft_consensus.cc:2747] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:45.575297 9825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.194:45923 (request call id 21) took 1083 ms. Trace:
I20250627 13:55:45.582072 9932 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:45.586825 9932 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:45.587733 9932 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:45.589087 9303 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1285 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:45.590687 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1285 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:45.596601 9760 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:45.605988 9932 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:46.000936 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250627 13:55:45.578294 9825 rpcz_store.cc:276] 0627 13:55:44.491454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:44.491516 (+ 62us) service_pool.cc:225] Handling call
0627 13:55:44.577139 (+ 85623us) raft_consensus.cc:1469] Updating replica for 195 ops
0627 13:55:44.667707 (+ 90568us) raft_consensus.cc:1532] Early marking committed up to index 1090
0627 13:55:44.667864 (+ 157us) raft_consensus.cc:1537] Triggering prepare for 195 ops
0627 13:55:44.668020 (+ 156us) write_op.cc:270] Start()
0627 13:55:44.668049 (+ 29us) write_op.cc:276] Timestamp: P: 1751032540727032 usec, L: 0
0627 13:55:44.669281 (+ 1232us) write_op.cc:270] Start()
0627 13:55:44.669305 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540766948 usec, L: 0
0627 13:55:44.676308 (+ 7003us) write_op.cc:270] Start()
0627 13:55:44.676336 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540770596 usec, L: 0
0627 13:55:44.680215 (+ 3879us) write_op.cc:270] Start()
0627 13:55:44.680241 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540811701 usec, L: 0
0627 13:55:44.687877 (+ 7636us) write_op.cc:270] Start()
0627 13:55:44.687897 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540813042 usec, L: 0
0627 13:55:44.688085 (+ 188us) write_op.cc:270] Start()
0627 13:55:44.688101 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540843189 usec, L: 0
0627 13:55:44.688255 (+ 154us) write_op.cc:270] Start()
0627 13:55:44.688290 (+ 35us) write_op.cc:276] Timestamp: P: 1751032540847120 usec, L: 0
0627 13:55:44.688451 (+ 161us) write_op.cc:270] Start()
0627 13:55:44.688472 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540866373 usec, L: 0
0627 13:55:44.688650 (+ 178us) write_op.cc:270] Start()
0627 13:55:44.688672 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540883977 usec, L: 0
0627 13:55:44.715874 (+ 27202us) write_op.cc:270] Start()
0627 13:55:44.715901 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540899031 usec, L: 0
0627 13:55:44.719989 (+ 4088us) write_op.cc:270] Start()
0627 13:55:44.720016 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540922904 usec, L: 0
0627 13:55:44.746074 (+ 26058us) write_op.cc:270] Start()
0627 13:55:44.746104 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540935669 usec, L: 0
0627 13:55:44.773564 (+ 27460us) write_op.cc:270] Start()
0627 13:55:44.773595 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540936398 usec, L: 0
0627 13:55:44.773825 (+ 230us) write_op.cc:270] Start()
0627 13:55:44.773847 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540953330 usec, L: 0
0627 13:55:44.773989 (+ 142us) write_op.cc:270] Start()
0627 13:55:44.774007 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540986855 usec, L: 0
0627 13:55:44.774184 (+ 177us) write_op.cc:270] Start()
0627 13:55:44.774203 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541005908 usec, L: 0
0627 13:55:44.774358 (+ 155us) write_op.cc:270] Start()
0627 13:55:44.774379 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541017075 usec, L: 0
0627 13:55:44.774561 (+ 182us) write_op.cc:270] Start()
0627 13:55:44.774579 (+ 18us) write_op.cc:276] Timestamp: P: 1751032541028657 usec, L: 0
0627 13:55:44.774717 (+ 138us) write_op.cc:270] Start()
0627 13:55:44.774736 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541064996 usec, L: 0
0627 13:55:44.774868 (+ 132us) write_op.cc:270] Start()
0627 13:55:44.774887 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541078217 usec, L: 0
0627 13:55:44.775064 (+ 177us) write_op.cc:270] Start()
0627 13:55:44.775084 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541081832 usec, L: 0
0627 13:55:44.777585 (+ 2501us) write_op.cc:270] Start()
0627 13:55:44.777609 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541105684 usec, L: 0
0627 13:55:44.784031 (+ 6422us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x60f000016d18. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243c29e818 00007f243efb1bc9 00007f243ef57cea 00007f243ef55507 00007f2442cb56ed 00007f243d17f3b6 00007f243d17f0e5 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95
0627 13:55:44.784157 (+ 126us) write_op.cc:270] Start()
0627 13:55:44.784181 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541142629 usec, L: 0
0627 13:55:44.784331 (+ 150us) write_op.cc:270] Start()
0627 13:55:44.784360 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541168173 usec, L: 0
0627 13:55:44.786784 (+ 2424us) write_op.cc:270] Start()
0627 13:55:44.786809 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541188102 usec, L: 0
0627 13:55:44.787181 (+ 372us) write_op.cc:270] Start()
0627 13:55:44.787206 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541195056 usec, L: 0
0627 13:55:44.801403 (+ 14197us) write_op.cc:270] Start()
0627 13:55:44.801431 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541218473 usec, L: 0
0627 13:55:44.802908 (+ 1477us) write_op.cc:270] Start()
0627 13:55:44.802934 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541224696 usec, L: 0
0627 13:55:44.803146 (+ 212us) write_op.cc:270] Start()
0627 13:55:44.803170 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541269245 usec, L: 0
0627 13:55:44.803342 (+ 172us) write_op.cc:270] Start()
0627 13:55:44.803364 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541295458 usec, L: 0
0627 13:55:44.804609 (+ 1245us) write_op.cc:270] Start()
0627 13:55:44.804633 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541303321 usec, L: 0
0627 13:55:44.804789 (+ 156us) write_op.cc:270] Start()
0627 13:55:44.804810 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541311149 usec, L: 0
0627 13:55:44.807443 (+ 2633us) write_op.cc:270] Start()
0627 13:55:44.807469 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541319361 usec, L: 0
0627 13:55:44.811369 (+ 3900us) write_op.cc:270] Start()
0627 13:55:44.811398 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541333725 usec, L: 0
0627 13:55:44.814940 (+ 3542us) write_op.cc:270] Start()
0627 13:55:44.814965 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541358212 usec, L: 0
0627 13:55:44.819846 (+ 4881us) write_op.cc:270] Start()
0627 13:55:44.819874 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541399454 usec, L: 0
0627 13:55:44.820055 (+ 181us) write_op.cc:270] Start()
0627 13:55:44.820078 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541405663 usec, L: 0
0627 13:55:44.830467 (+ 10389us) write_op.cc:270] Start()
0627 13:55:44.830494 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541428961 usec, L: 0
0627 13:55:44.860198 (+ 29704us) write_op.cc:270] Start()
0627 13:55:44.860225 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541435033 usec, L: 0
0627 13:55:44.860446 (+ 221us) write_op.cc:270] Start()
0627 13:55:44.860473 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541483974 usec, L: 0
0627 13:55:44.860639 (+ 166us) write_op.cc:270] Start()
0627 13:55:44.860661 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541488347 usec, L: 0
0627 13:55:44.862952 (+ 2291us) write_op.cc:270] Start()
0627 13:55:44.862976 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541493530 usec, L: 0
0627 13:55:44.864319 (+ 1343us) write_op.cc:270] Start()
0627 13:55:44.864349 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541504750 usec, L: 0
0627 13:55:44.874607 (+ 10258us) write_op.cc:270] Start()
0627 13:55:44.874641 (+ 34us) write_op.cc:276] Timestamp: P: 1751032541546811 usec, L: 0
0627 13:55:44.904527 (+ 29886us) write_op.cc:270] Start()
0627 13:55:44.904558 (+ 31us) write_op.cc:276] Timestamp: P: 1751032541576692 usec, L: 0
0627 13:55:44.904764 (+ 206us) write_op.cc:270] Start()
0627 13:55:44.904788 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541582733 usec, L: 0
0627 13:55:44.920916 (+ 16128us) write_op.cc:270] Start()
0627 13:55:44.920945 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541601715 usec, L: 0
0627 13:55:44.932476 (+ 11531us) write_op.cc:270] Start()
0627 13:55:44.932504 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541603443 usec, L: 0
0627 13:55:44.937321 (+ 4817us) write_op.cc:270] Start()
0627 13:55:44.937349 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541637404 usec, L: 0
0627 13:55:44.937538 (+ 189us) write_op.cc:270] Start()
0627 13:55:44.937563 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541642548 usec, L: 0
0627 13:55:44.937752 (+ 189us) write_op.cc:270] Start()
0627 13:55:44.937775 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541675479 usec, L: 0
0627 13:55:44.937937 (+ 162us) write_op.cc:270] Start()
0627 13:55:44.937958 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541699484 usec, L: 0
0627 13:55:44.939877 (+ 1919us) write_op.cc:270] Start()
0627 13:55:44.939904 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541700773 usec, L: 0
0627 13:55:44.942580 (+ 2676us) write_op.cc:270] Start()
0627 13:55:44.942610 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541705067 usec, L: 0
0627 13:55:44.944150 (+ 1540us) write_op.cc:270] Start()
0627 13:55:44.944757 (+ 607us) write_op.cc:276] Timestamp: P: 1751032541757231 usec, L: 0
0627 13:55:44.945557 (+ 800us) write_op.cc:270] Start()
0627 13:55:44.945581 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541794199 usec, L: 0
0627 13:55:44.947259 (+ 1678us) write_op.cc:270] Start()
0627 13:55:44.947276 (+ 17us) write_op.cc:276] Timestamp: P: 1751032541808697 usec, L: 0
0627 13:55:44.954985 (+ 7709us) write_op.cc:270] Start()
0627 13:55:44.955007 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541811494 usec, L: 0
0627 13:55:44.961959 (+ 6952us) write_op.cc:270] Start()
0627 13:55:44.961989 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541846051 usec, L: 0
0627 13:55:44.962200 (+ 211us) write_op.cc:270] Start()
0627 13:55:44.962222 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541874350 usec, L: 0
0627 13:55:44.962393 (+ 171us) write_op.cc:270] Start()
0627 13:55:44.962415 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541875245 usec, L: 0
0627 13:55:44.962554 (+ 139us) write_op.cc:270] Start()
0627 13:55:44.962571 (+ 17us) write_op.cc:276] Timestamp: P: 1751032541883086 usec, L: 0
0627 13:55:44.963026 (+ 455us) write_op.cc:270] Start()
0627 13:55:44.963052 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541925055 usec, L: 0
0627 13:55:45.018884 (+ 55832us) write_op.cc:270] Start()
0627 13:55:45.018910 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541942091 usec, L: 0
0627 13:55:45.019110 (+ 200us) write_op.cc:270] Start()
0627 13:55:45.019129 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541951285 usec, L: 0
0627 13:55:45.019273 (+ 144us) write_op.cc:270] Start()
0627 13:55:45.019297 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541965695 usec, L: 0
0627 13:55:45.019483 (+ 186us) write_op.cc:270] Start()
0627 13:55:45.019508 (+ 25us) write_op.cc:276] Timestamp: P: 1751032542010521 usec, L: 0
0627 13:55:45.019690 (+ 182us) write_op.cc:270] Start()
0627 13:55:45.019711 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542061074 usec, L: 0
0627 13:55:45.051541 (+ 31830us) write_op.cc:270] Start()
0627 13:55:45.051568 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542068991 usec, L: 0
0627 13:55:45.051756 (+ 188us) write_op.cc:270] Start()
0627 13:55:45.051774 (+ 18us) write_op.cc:276] Timestamp: P: 1751032542100458 usec, L: 0
0627 13:55:45.051894 (+ 120us) write_op.cc:270] Start()
0627 13:55:45.051913 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542122830 usec, L: 0
0627 13:55:45.052051 (+ 138us) write_op.cc:270] Start()
0627 13:55:45.052071 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542126802 usec, L: 0
0627 13:55:45.052236 (+ 165us) write_op.cc:270] Start()
0627 13:55:45.052256 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542131404 usec, L: 0
0627 13:55:45.052381 (+ 125us) write_op.cc:270] Start()
0627 13:55:45.052410 (+ 29us) write_op.cc:276] Timestamp: P: 1751032542163084 usec, L: 0
0627 13:55:45.052553 (+ 143us) write_op.cc:270] Start()
0627 13:55:45.052572 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542175090 usec, L: 0
0627 13:55:45.052716 (+ 144us) write_op.cc:270] Start()
0627 13:55:45.052735 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542180267 usec, L: 0
0627 13:55:45.052881 (+ 146us) write_op.cc:270] Start()
0627 13:55:45.052900 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542216236 usec, L: 0
0627 13:55:45.053069 (+ 169us) write_op.cc:270] Start()
0627 13:55:45.053089 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542228164 usec, L: 0
0627 13:55:45.053227 (+ 138us) write_op.cc:270] Start()
0627 13:55:45.053246 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542231597 usec, L: 0
0627 13:55:45.053405 (+ 159us) write_op.cc:270] Start()
0627 13:55:45.053424 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542259839 usec, L: 0
0627 13:55:45.053576 (+ 152us) write_op.cc:270] Start()
0627 13:55:45.053595 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542262999 usec, L: 0
0627 13:55:45.053755 (+ 160us) write_op.cc:270] Start()
0627 13:55:45.053775 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542294291 usec, L: 0
0627 13:55:45.053916 (+ 141us) write_op.cc:270] Start()
0627 13:55:45.053935 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542323547 usec, L: 0
0627 13:55:45.056136 (+ 2201us) write_op.cc:270] Start()
0627 13:55:45.056159 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542345617 usec, L: 0
0627 13:55:45.074984 (+ 18825us) write_op.cc:270] Start()
0627 13:55:45.075014 (+ 30us) write_op.cc:276] Timestamp: P: 1751032542356896 usec, L: 0
0627 13:55:45.075197 (+ 183us) write_op.cc:270] Start()
0627 13:55:45.075215 (+ 18us) write_op.cc:276] Timestamp: P: 1751032542404530 usec, L: 0
0627 13:55:45.075333 (+ 118us) write_op.cc:270] Start()
0627 13:55:45.075349 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542409682 usec, L: 0
0627 13:55:45.075462 (+ 113us) write_op.cc:270] Start()
0627 13:55:45.075479 (+ 17us) write_op.cc:276] Timestamp: P: 1751032542431263 usec, L: 0
0627 13:55:45.075580 (+ 101us) write_op.cc:270] Start()
0627 13:55:45.075594 (+ 14us) write_op.cc:276] Timestamp: P: 1751032542433588 usec, L: 0
0627 13:55:45.075808 (+ 214us) write_op.cc:270] Start()
0627 13:55:45.075826 (+ 18us) write_op.cc:276] Timestamp: P: 1751032542445319 usec, L: 0
0627 13:55:45.075960 (+ 134us) write_op.cc:270] Start()
0627 13:55:45.075977 (+ 17us) write_op.cc:276] Timestamp: P: 1751032542446295 usec, L: 0
0627 13:55:45.076091 (+ 114us) write_op.cc:270] Start()
0627 13:55:45.076106 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542476991 usec, L: 0
0627 13:55:45.076228 (+ 122us) write_op.cc:270] Start()
0627 13:55:45.076244 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542530489 usec, L: 0
0627 13:55:45.076343 (+ 99us) write_op.cc:270] Start()
0627 13:55:45.076360 (+ 17us) write_op.cc:276] Timestamp: P: 1751032542531681 usec, L: 0
0627 13:55:45.076481 (+ 121us) write_op.cc:270] Start()
0627 13:55:45.076497 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542565218 usec, L: 0
0627 13:55:45.076608 (+ 111us) write_op.cc:270] Start()
0627 13:55:45.076625 (+ 17us) write_op.cc:276] Timestamp: P: 1751032542576301 usec, L: 0
0627 13:55:45.076726 (+ 101us) write_op.cc:270] Start()
0627 13:55:45.076741 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542590082 usec, L: 0
0627 13:55:45.076860 (+ 119us) write_op.cc:270] Start()
0627 13:55:45.076875 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542630777 usec, L: 0
0627 13:55:45.076980 (+ 105us) write_op.cc:270] Start()
0627 13:55:45.076994 (+ 14us) write_op.cc:276] Timestamp: P: 1751032542643156 usec, L: 0
0627 13:55:45.077093 (+ 99us) write_op.cc:270] Start()
0627 13:55:45.077109 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542648522 usec, L: 0
0627 13:55:45.077226 (+ 117us) write_op.cc:270] Start()
0627 13:55:45.077241 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542687725 usec, L: 0
0627 13:55:45.077342 (+ 101us) write_op.cc:270] Start()
0627 13:55:45.077364 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542698160 usec, L: 0
0627 13:55:45.077479 (+ 115us) write_op.cc:270] Start()
0627 13:55:45.077494 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542730519 usec, L: 0
0627 13:55:45.077650 (+ 156us) write_op.cc:270] Start()
0627 13:55:45.077666 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542743300 usec, L: 0
0627 13:55:45.077771 (+ 105us) write_op.cc:270] Start()
0627 13:55:45.077786 (+ 15us) write_op.cc:276] Timestamp: P: 1751032542743866 usec, L: 0
0627 13:55:45.077893 (+ 107us) write_op.cc:270] Start()
0627 13:55:45.077909 (+ 16us) write_op.cc:276] Timestamp: P: 1751032542763213 usec, L: 0
0627 13:55:45.078019 (+ 110us) write_op.cc:270] Start()
0627 13:55:45.088544 (+ 10525us) write_op.cc:276] Timestamp: P: 1751032542791586 usec, L: 0
0627 13:55:45.099732 (+ 11188us) write_op.cc:270] Start()
0627 13:55:45.099760 (+ 28us) write_op.cc:276] Timestamp: P: 1751032542806207 usec, L: 0
0627 13:55:45.099943 (+ 183us) write_op.cc:270] Start()
0627 13:55:45.099964 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542816265 usec, L: 0
0627 13:55:45.100109 (+ 145us) write_op.cc:270] Start()
0627 13:55:45.100129 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542826125 usec, L: 0
0627 13:55:45.100285 (+ 156us) write_op.cc:270] Start()
0627 13:55:45.100306 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542861322 usec, L: 0
0627 13:55:45.100497 (+ 191us) write_op.cc:270] Start()
0627 13:55:45.100520 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542888872 usec, L: 0
0627 13:55:45.100686 (+ 166us) write_op.cc:270] Start()
0627 13:55:45.100706 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542895373 usec, L: 0
0627 13:55:45.100845 (+ 139us) write_op.cc:270] Start()
0627 13:55:45.100864 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542906893 usec, L: 0
0627 13:55:45.103460 (+ 2596us) write_op.cc:270] Start()
0627 13:55:45.103486 (+ 26us) write_op.cc:276] Timestamp: P: 1751032542946182 usec, L: 0
0627 13:55:45.103690 (+ 204us) write_op.cc:270] Start()
0627 13:55:45.103712 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542961251 usec, L: 0
0627 13:55:45.103879 (+ 167us) write_op.cc:270] Start()
0627 13:55:45.103898 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542967727 usec, L: 0
0627 13:55:45.104049 (+ 151us) write_op.cc:270] Start()
0627 13:55:45.104075 (+ 26us) write_op.cc:276] Timestamp: P: 1751032543026110 usec, L: 0
0627 13:55:45.104216 (+ 141us) write_op.cc:270] Start()
0627 13:55:45.104236 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543026740 usec, L: 0
0627 13:55:45.104371 (+ 135us) write_op.cc:270] Start()
0627 13:55:45.104390 (+ 19us) write_op.cc:276] Timestamp: P: 1751032543054077 usec, L: 0
0627 13:55:45.105073 (+ 683us) write_op.cc:270] Start()
0627 13:55:45.105100 (+ 27us) write_op.cc:276] Timestamp: P: 1751032543059033 usec, L: 0
0627 13:55:45.105299 (+ 199us) write_op.cc:270] Start()
0627 13:55:45.105322 (+ 23us) write_op.cc:276] Timestamp: P: 1751032543122355 usec, L: 0
0627 13:55:45.105485 (+ 163us) write_op.cc:270] Start()
0627 13:55:45.105507 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543123275 usec, L: 0
0627 13:55:45.123997 (+ 18490us) write_op.cc:270] Start()
0627 13:55:45.124019 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543142665 usec, L: 0
0627 13:55:45.127732 (+ 3713us) write_op.cc:270] Start()
0627 13:55:45.127761 (+ 29us) write_op.cc:276] Timestamp: P: 1751032543169037 usec, L: 0
0627 13:55:45.127928 (+ 167us) write_op.cc:270] Start()
0627 13:55:45.129506 (+ 1578us) write_op.cc:276] Timestamp: P: 1751032543170451 usec, L: 0
0627 13:55:45.130822 (+ 1316us) write_op.cc:270] Start()
0627 13:55:45.130844 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543195916 usec, L: 0
0627 13:55:45.130997 (+ 153us) write_op.cc:270] Start()
0627 13:55:45.131014 (+ 17us) write_op.cc:276] Timestamp: P: 1751032543205746 usec, L: 0
0627 13:55:45.131153 (+ 139us) write_op.cc:270] Start()
0627 13:55:45.131170 (+ 17us) write_op.cc:276] Timestamp: P: 1751032543215396 usec, L: 0
0627 13:55:45.131287 (+ 117us) write_op.cc:270] Start()
0627 13:55:45.131304 (+ 17us) write_op.cc:276] Timestamp: P: 1751032543227482 usec, L: 0
0627 13:55:45.131431 (+ 127us) write_op.cc:270] Start()
0627 13:55:45.131447 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543258156 usec, L: 0
0627 13:55:45.131547 (+ 100us) write_op.cc:270] Start()
0627 13:55:45.131562 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543266880 usec, L: 0
0627 13:55:45.131702 (+ 140us) write_op.cc:270] Start()
0627 13:55:45.131718 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543284490 usec, L: 0
0627 13:55:45.131831 (+ 113us) write_op.cc:270] Start()
0627 13:55:45.131846 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543299662 usec, L: 0
0627 13:55:45.131955 (+ 109us) write_op.cc:270] Start()
0627 13:55:45.131971 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543327617 usec, L: 0
0627 13:55:45.132086 (+ 115us) write_op.cc:270] Start()
0627 13:55:45.132102 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543337011 usec, L: 0
0627 13:55:45.132213 (+ 111us) write_op.cc:270] Start()
0627 13:55:45.132228 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543345736 usec, L: 0
0627 13:55:45.132338 (+ 110us) write_op.cc:270] Start()
0627 13:55:45.132353 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543356275 usec, L: 0
0627 13:55:45.132482 (+ 129us) write_op.cc:270] Start()
0627 13:55:45.132512 (+ 30us) write_op.cc:276] Timestamp: P: 1751032543377190 usec, L: 0
0627 13:55:45.132624 (+ 112us) write_op.cc:270] Start()
0627 13:55:45.132640 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543405676 usec, L: 0
0627 13:55:45.134945 (+ 2305us) write_op.cc:270] Start()
0627 13:55:45.134965 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543406901 usec, L: 0
0627 13:55:45.135093 (+ 128us) write_op.cc:270] Start()
0627 13:55:45.135111 (+ 18us) write_op.cc:276] Timestamp: P: 1751032543425330 usec, L: 0
0627 13:55:45.135243 (+ 132us) write_op.cc:270] Start()
0627 13:55:45.135262 (+ 19us) write_op.cc:276] Timestamp: P: 1751032543433823 usec, L: 0
0627 13:55:45.136869 (+ 1607us) write_op.cc:270] Start()
0627 13:55:45.136888 (+ 19us) write_op.cc:276] Timestamp: P: 1751032543456097 usec, L: 0
0627 13:55:45.141765 (+ 4877us) write_op.cc:270] Start()
0627 13:55:45.141790 (+ 25us) write_op.cc:276] Timestamp: P: 1751032543477273 usec, L: 0
0627 13:55:45.143247 (+ 1457us) write_op.cc:270] Start()
0627 13:55:45.143271 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543501262 usec, L: 0
0627 13:55:45.143433 (+ 162us) write_op.cc:270] Start()
0627 13:55:45.143453 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543515420 usec, L: 0
0627 13:55:45.143579 (+ 126us) write_op.cc:270] Start()
0627 13:55:45.143597 (+ 18us) write_op.cc:276] Timestamp: P: 1751032543550878 usec, L: 0
0627 13:55:45.143716 (+ 119us) write_op.cc:270] Start()
0627 13:55:45.143732 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543577977 usec, L: 0
0627 13:55:45.143863 (+ 131us) write_op.cc:270] Start()
0627 13:55:45.143878 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543583349 usec, L: 0
0627 13:55:45.143985 (+ 107us) write_op.cc:270] Start()
0627 13:55:45.143999 (+ 14us) write_op.cc:276] Timestamp: P: 1751032543598879 usec, L: 0
0627 13:55:45.144106 (+ 107us) write_op.cc:270] Start()
0627 13:55:45.144120 (+ 14us) write_op.cc:276] Timestamp: P: 1751032543605726 usec, L: 0
0627 13:55:45.144221 (+ 101us) write_op.cc:270] Start()
0627 13:55:45.144239 (+ 18us) write_op.cc:276] Timestamp: P: 1751032543647460 usec, L: 0
0627 13:55:45.144394 (+ 155us) write_op.cc:270] Start()
0627 13:55:45.144411 (+ 17us) write_op.cc:276] Timestamp: P: 1751032543688333 usec, L: 0
0627 13:55:45.144523 (+ 112us) write_op.cc:270] Start()
0627 13:55:45.144564 (+ 41us) write_op.cc:276] Timestamp: P: 1751032543688955 usec, L: 0
0627 13:55:45.144678 (+ 114us) write_op.cc:270] Start()
0627 13:55:45.144695 (+ 17us) write_op.cc:276] Timestamp: P: 1751032543691544 usec, L: 0
0627 13:55:45.189836 (+ 45141us) write_op.cc:270] Start()
0627 13:55:45.189869 (+ 33us) write_op.cc:276] Timestamp: P: 1751032543715686 usec, L: 0
0627 13:55:45.198073 (+ 8204us) write_op.cc:270] Start()
0627 13:55:45.198108 (+ 35us) write_op.cc:276] Timestamp: P: 1751032543756223 usec, L: 0
0627 13:55:45.209159 (+ 11051us) write_op.cc:270] Start()
0627 13:55:45.209191 (+ 32us) write_op.cc:276] Timestamp: P: 1751032543766426 usec, L: 0
0627 13:55:45.211584 (+ 2393us) write_op.cc:270] Start()
0627 13:55:45.211605 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543767673 usec, L: 0
0627 13:55:45.211739 (+ 134us) write_op.cc:270] Start()
0627 13:55:45.211754 (+ 15us) write_op.cc:276] Timestamp: P: 1751032543779132 usec, L: 0
0627 13:55:45.232246 (+ 20492us) write_op.cc:270] Start()
0627 13:55:45.232277 (+ 31us) write_op.cc:276] Timestamp: P: 1751032543812561 usec, L: 0
0627 13:55:45.238088 (+ 5811us) write_op.cc:270] Start()
0627 13:55:45.238118 (+ 30us) write_op.cc:276] Timestamp: P: 1751032543822845 usec, L: 0
0627 13:55:45.245704 (+ 7586us) write_op.cc:270] Start()
0627 13:55:45.245730 (+ 26us) write_op.cc:276] Timestamp: P: 1751032543847980 usec, L: 0
0627 13:55:45.245916 (+ 186us) write_op.cc:270] Start()
0627 13:55:45.245944 (+ 28us) write_op.cc:276] Timestamp: P: 1751032543869697 usec, L: 0
0627 13:55:45.279658 (+ 33714us) write_op.cc:270] Start()
0627 13:55:45.279690 (+ 32us) write_op.cc:276] Timestamp: P: 1751032543900093 usec, L: 0
0627 13:55:45.279872 (+ 182us) write_op.cc:270] Start()
0627 13:55:45.279890 (+ 18us) write_op.cc:276] Timestamp: P: 1751032543922201 usec, L: 0
0627 13:55:45.284653 (+ 4763us) write_op.cc:270] Start()
0627 13:55:45.284682 (+ 29us) write_op.cc:276] Timestamp: P: 1751032543934786 usec, L: 0
0627 13:55:45.287252 (+ 2570us) write_op.cc:270] Start()
0627 13:55:45.287278 (+ 26us) write_op.cc:276] Timestamp: P: 1751032543953382 usec, L: 0
0627 13:55:45.287449 (+ 171us) write_op.cc:270] Start()
0627 13:55:45.287471 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543976790 usec, L: 0
0627 13:55:45.310318 (+ 22847us) write_op.cc:270] Start()
0627 13:55:45.317948 (+ 7630us) write_op.cc:276] Timestamp: P: 1751032543992088 usec, L: 0
0627 13:55:45.320959 (+ 3011us) write_op.cc:270] Start()
0627 13:55:45.320982 (+ 23us) write_op.cc:276] Timestamp: P: 1751032544015849 usec, L: 0
0627 13:55:45.326818 (+ 5836us) write_op.cc:270] Start()
0627 13:55:45.326847 (+ 29us) write_op.cc:276] Timestamp: P: 1751032544036830 usec, L: 0
0627 13:55:45.333469 (+ 6622us) write_op.cc:270] Start()
0627 13:55:45.333499 (+ 30us) write_op.cc:276] Timestamp: P: 1751032544059740 usec, L: 0
0627 13:55:45.337867 (+ 4368us) write_op.cc:270] Start()
0627 13:55:45.337897 (+ 30us) write_op.cc:276] Timestamp: P: 1751032544061624 usec, L: 0
0627 13:55:45.344032 (+ 6135us) write_op.cc:270] Start()
0627 13:55:45.344060 (+ 28us) write_op.cc:276] Timestamp: P: 1751032544091576 usec, L: 0
0627 13:55:45.363787 (+ 19727us) write_op.cc:270] Start()
0627 13:55:45.363813 (+ 26us) write_op.cc:276] Timestamp: P: 1751032544096606 usec, L: 0
0627 13:55:45.363978 (+ 165us) write_op.cc:270] Start()
0627 13:55:45.364005 (+ 27us) write_op.cc:276] Timestamp: P: 1751032544127710 usec, L: 0
0627 13:55:45.364131 (+ 126us) write_op.cc:270] Start()
0627 13:55:45.364146 (+ 15us) write_op.cc:276] Timestamp: P: 1751032544133047 usec, L: 0
0627 13:55:45.364285 (+ 139us) write_op.cc:270] Start()
0627 13:55:45.364305 (+ 20us) write_op.cc:276] Timestamp: P: 1751032544163611 usec, L: 0
0627 13:55:45.364448 (+ 143us) write_op.cc:270] Start()
0627 13:55:45.364478 (+ 30us) write_op.cc:276] Timestamp: P: 1751032544181949 usec, L: 0
0627 13:55:45.364633 (+ 155us) write_op.cc:270] Start()
0627 13:55:45.364653 (+ 20us) write_op.cc:276] Timestamp: P: 1751032544198524 usec, L: 0
0627 13:55:45.364787 (+ 134us) write_op.cc:270] Start()
0627 13:55:45.364811 (+ 24us) write_op.cc:276] Timestamp: P: 1751032544200423 usec, L: 0
0627 13:55:45.364962 (+ 151us) write_op.cc:270] Start()
0627 13:55:45.364981 (+ 19us) write_op.cc:276] Timestamp: P: 1751032544288452 usec, L: 0
0627 13:55:45.365138 (+ 157us) write_op.cc:270] Start()
0627 13:55:45.365159 (+ 21us) write_op.cc:276] Timestamp: P: 1751032544298787 usec, L: 0
0627 13:55:45.365303 (+ 144us) write_op.cc:270] Start()
0627 13:55:45.365322 (+ 19us) write_op.cc:276] Timestamp: P: 1751032544303835 usec, L: 0
0627 13:55:45.368117 (+ 2795us) write_op.cc:270] Start()
0627 13:55:45.368161 (+ 44us) write_op.cc:276] Timestamp: P: 1751032544309047 usec, L: 0
0627 13:55:45.368341 (+ 180us) write_op.cc:270] Start()
0627 13:55:45.368364 (+ 23us) write_op.cc:276] Timestamp: P: 1751032544327266 usec, L: 0
0627 13:55:45.368555 (+ 191us) write_op.cc:270] Start()
0627 13:55:45.368580 (+ 25us) write_op.cc:276] Timestamp: P: 1751032544341835 usec, L: 0
0627 13:55:45.368733 (+ 153us) write_op.cc:270] Start()
0627 13:55:45.368756 (+ 23us) write_op.cc:276] Timestamp: P: 1751032544361966 usec, L: 0
0627 13:55:45.368910 (+ 154us) write_op.cc:270] Start()
0627 13:55:45.368931 (+ 21us) write_op.cc:276] Timestamp: P: 1751032544398585 usec, L: 0
0627 13:55:45.392755 (+ 23824us) write_op.cc:270] Start()
0627 13:55:45.392782 (+ 27us) write_op.cc:276] Timestamp: P: 1751032544402270 usec, L: 0
0627 13:55:45.392946 (+ 164us) write_op.cc:270] Start()
0627 13:55:45.392965 (+ 19us) write_op.cc:276] Timestamp: P: 1751032544426737 usec, L: 0
0627 13:55:45.393075 (+ 110us) write_op.cc:270] Start()
0627 13:55:45.393094 (+ 19us) write_op.cc:276] Timestamp: P: 1751032544429293 usec, L: 0
0627 13:55:45.393245 (+ 151us) write_op.cc:270]
I20250627 13:55:46.020035 9932 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:46.022686 9932 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:46.023432 9932 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:46.026230 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1285 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:46.026321 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1285 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:46.033428 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:46.035750 9932 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:46.134227 9764 log.cc:927] Time spent T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Append to log took a long time: real 0.080s user 0.004s sys 0.000s
I20250627 13:55:46.371062 9932 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:55:46.376070 9932 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:46.376794 9932 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:46.380120 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1368 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:46.487787 9171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "22aa1c36043f4e8781bac641bd1035aa" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1368 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:46.497608 9920 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:46.511942 9787 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57736 (request call id 48) took 2048 ms. Trace:
I20250627 13:55:46.512102 9787 rpcz_store.cc:276] 0627 13:55:44.458073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:44.458131 (+ 58us) service_pool.cc:225] Handling call
0627 13:55:46.493839 (+2035708us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x616000018d54. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:46.506278 (+ 12439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4678820736}
I20250627 13:55:46.542729 9827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 30) took 6417 ms. Trace:
I20250627 13:55:46.604360 9761 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:46.604809 9920 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:46.721742 9764 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap replayed 1/1 log segments. Stats: ops{read=158 overwritten=0 applied=156 ignored=0} inserts{seen=1273 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250627 13:55:46.768297 9764 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Bootstrap complete.
I20250627 13:55:46.769632 9764 ts_tablet_manager.cc:1397] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Time spent bootstrapping tablet: real 6.534s user 0.306s sys 0.062s
I20250627 13:55:46.774673 9764 raft_consensus.cc:357] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:46.798355 9764 raft_consensus.cc:738] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6bddc499b4363a95de5d2578e4a07, State: Initialized, Role: FOLLOWER
I20250627 13:55:46.798913 9764 consensus_queue.cc:260] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.158, Last appended by leader: 158, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:46.799723 9764 ts_tablet_manager.cc:1428] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250627 13:55:46.918083 9920 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:46.918277 9920 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:46.952171 9920 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:46.954332 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:46.963171 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:46.965575 9761 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:46.977619 9920 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:55:47.121022 9920 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:55:47.121228 9920 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:47.122701 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 158 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:47.124044 9920 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:47.126634 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 158 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:47.127264 9761 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:47.127656 9920 raft_consensus.cc:2747] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:47.461483 9920 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:47.461701 9920 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:47.463063 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:47.464912 9920 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:47.471304 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:47.484874 9761 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:47.497668 9940 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:55:47.844229 9830 raft_consensus.cc:1215] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.156->[1.157-1.1383] Dedup: 1.158->[1.159-1.1383]
I20250627 13:55:47.993737 9940 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:47.996337 9940 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:47.998595 9940 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:47.999320 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:47.999979 9173 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1055 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:48.002455 9760 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:48.003593 9940 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:55:46.555485 9827 rpcz_store.cc:276] 0627 13:55:40.101591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:40.101649 (+ 58us) service_pool.cc:225] Handling call
0627 13:55:40.573808 (+472159us) raft_consensus.cc:1469] Updating replica for 895 ops
0627 13:55:41.323419 (+749611us) raft_consensus.cc:1532] Early marking committed up to index 160
0627 13:55:41.323539 (+ 120us) raft_consensus.cc:1537] Triggering prepare for 895 ops
0627 13:55:41.327179 (+ 3640us) write_op.cc:270] Start()
0627 13:55:41.327218 (+ 39us) write_op.cc:276] Timestamp: P: 1751032522976216 usec, L: 0
0627 13:55:41.508213 (+180995us) write_op.cc:270] Start()
0627 13:55:41.508240 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523013230 usec, L: 0
0627 13:55:41.508434 (+ 194us) write_op.cc:270] Start()
0627 13:55:41.508454 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523039944 usec, L: 0
0627 13:55:41.515859 (+ 7405us) write_op.cc:270] Start()
0627 13:55:41.515890 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523040686 usec, L: 0
0627 13:55:41.518994 (+ 3104us) write_op.cc:270] Start()
0627 13:55:41.519030 (+ 36us) write_op.cc:276] Timestamp: P: 1751032523042346 usec, L: 0
0627 13:55:41.549672 (+ 30642us) write_op.cc:270] Start()
0627 13:55:41.549718 (+ 46us) write_op.cc:276] Timestamp: P: 1751032523043487 usec, L: 0
0627 13:55:41.590739 (+ 41021us) write_op.cc:270] Start()
0627 13:55:41.590794 (+ 55us) write_op.cc:276] Timestamp: P: 1751032523074658 usec, L: 0
0627 13:55:41.610241 (+ 19447us) write_op.cc:270] Start()
0627 13:55:41.610274 (+ 33us) write_op.cc:276] Timestamp: P: 1751032523086424 usec, L: 0
0627 13:55:41.610560 (+ 286us) write_op.cc:270] Start()
0627 13:55:41.610585 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523120540 usec, L: 0
0627 13:55:41.610782 (+ 197us) write_op.cc:270] Start()
0627 13:55:41.610803 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523133990 usec, L: 0
0627 13:55:41.610968 (+ 165us) write_op.cc:270] Start()
0627 13:55:41.610988 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523134915 usec, L: 0
0627 13:55:41.611133 (+ 145us) write_op.cc:270] Start()
0627 13:55:41.611152 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523138185 usec, L: 0
0627 13:55:41.611318 (+ 166us) write_op.cc:270] Start()
0627 13:55:41.611339 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523156944 usec, L: 0
0627 13:55:41.611474 (+ 135us) write_op.cc:270] Start()
0627 13:55:41.611493 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523180772 usec, L: 0
0627 13:55:41.611646 (+ 153us) write_op.cc:270] Start()
0627 13:55:41.611673 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523189042 usec, L: 0
0627 13:55:41.611861 (+ 188us) write_op.cc:270] Start()
0627 13:55:41.611883 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523193463 usec, L: 0
0627 13:55:41.612035 (+ 152us) write_op.cc:270] Start()
0627 13:55:41.612056 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523208480 usec, L: 0
0627 13:55:41.612191 (+ 135us) write_op.cc:270] Start()
0627 13:55:41.612209 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523226466 usec, L: 0
0627 13:55:41.612372 (+ 163us) write_op.cc:270] Start()
0627 13:55:41.612391 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523238821 usec, L: 0
0627 13:55:41.612545 (+ 154us) write_op.cc:270] Start()
0627 13:55:41.612564 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523249593 usec, L: 0
0627 13:55:41.612712 (+ 148us) write_op.cc:270] Start()
0627 13:55:41.612743 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523255896 usec, L: 0
0627 13:55:41.612895 (+ 152us) write_op.cc:270] Start()
0627 13:55:41.612916 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523290838 usec, L: 0
0627 13:55:41.613047 (+ 131us) write_op.cc:270] Start()
0627 13:55:41.613066 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523301244 usec, L: 0
0627 13:55:41.613186 (+ 120us) write_op.cc:270] Start()
0627 13:55:41.613203 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523314252 usec, L: 0
0627 13:55:41.617878 (+ 4675us) write_op.cc:270] Start()
0627 13:55:41.617898 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523322721 usec, L: 0
0627 13:55:41.618214 (+ 316us) write_op.cc:270] Start()
0627 13:55:41.618236 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523326116 usec, L: 0
0627 13:55:41.619177 (+ 941us) write_op.cc:270] Start()
0627 13:55:41.619205 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523346558 usec, L: 0
0627 13:55:41.629095 (+ 9890us) write_op.cc:270] Start()
0627 13:55:41.629127 (+ 32us) write_op.cc:276] Timestamp: P: 1751032523376525 usec, L: 0
0627 13:55:41.646341 (+ 17214us) write_op.cc:270] Start()
0627 13:55:41.646376 (+ 35us) write_op.cc:276] Timestamp: P: 1751032523390466 usec, L: 0
0627 13:55:41.663749 (+ 17373us) write_op.cc:270] Start()
0627 13:55:41.676503 (+ 12754us) write_op.cc:276] Timestamp: P: 1751032523392082 usec, L: 0
0627 13:55:41.677517 (+ 1014us) write_op.cc:270] Start()
0627 13:55:41.677548 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523395571 usec, L: 0
0627 13:55:41.681571 (+ 4023us) write_op.cc:270] Start()
0627 13:55:41.681597 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523427091 usec, L: 0
0627 13:55:41.685034 (+ 3437us) write_op.cc:270] Start()
0627 13:55:41.685068 (+ 34us) write_op.cc:276] Timestamp: P: 1751032523455227 usec, L: 0
0627 13:55:41.686591 (+ 1523us) write_op.cc:270] Start()
0627 13:55:41.686623 (+ 32us) write_op.cc:276] Timestamp: P: 1751032523458374 usec, L: 0
0627 13:55:41.708372 (+ 21749us) write_op.cc:270] Start()
0627 13:55:41.713248 (+ 4876us) write_op.cc:276] Timestamp: P: 1751032523459654 usec, L: 0
0627 13:55:41.713482 (+ 234us) write_op.cc:270] Start()
0627 13:55:41.713507 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523463179 usec, L: 0
0627 13:55:41.713662 (+ 155us) write_op.cc:270] Start()
0627 13:55:41.713683 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523491215 usec, L: 0
0627 13:55:41.713875 (+ 192us) write_op.cc:270] Start()
0627 13:55:41.713903 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523507915 usec, L: 0
0627 13:55:41.714162 (+ 259us) write_op.cc:270] Start()
0627 13:55:41.714197 (+ 35us) write_op.cc:276] Timestamp: P: 1751032523523654 usec, L: 0
0627 13:55:41.714455 (+ 258us) write_op.cc:270] Start()
0627 13:55:41.714491 (+ 36us) write_op.cc:276] Timestamp: P: 1751032523541280 usec, L: 0
0627 13:55:41.714786 (+ 295us) write_op.cc:270] Start()
0627 13:55:41.714816 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523542012 usec, L: 0
0627 13:55:41.715019 (+ 203us) write_op.cc:270] Start()
0627 13:55:41.715047 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523575075 usec, L: 0
0627 13:55:41.715191 (+ 144us) write_op.cc:270] Start()
0627 13:55:41.715209 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523577111 usec, L: 0
0627 13:55:41.716974 (+ 1765us) write_op.cc:270] Start()
0627 13:55:41.716999 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523605673 usec, L: 0
0627 13:55:41.736486 (+ 19487us) write_op.cc:270] Start()
0627 13:55:41.736527 (+ 41us) write_op.cc:276] Timestamp: P: 1751032523606482 usec, L: 0
0627 13:55:41.764066 (+ 27539us) write_op.cc:270] Start()
0627 13:55:41.769185 (+ 5119us) write_op.cc:276] Timestamp: P: 1751032523618997 usec, L: 0
0627 13:55:41.773189 (+ 4004us) write_op.cc:270] Start()
0627 13:55:41.773220 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523626959 usec, L: 0
0627 13:55:41.773411 (+ 191us) write_op.cc:270] Start()
0627 13:55:41.773432 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523663947 usec, L: 0
0627 13:55:41.773576 (+ 144us) write_op.cc:270] Start()
0627 13:55:41.773601 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523666417 usec, L: 0
0627 13:55:41.773738 (+ 137us) write_op.cc:270] Start()
0627 13:55:41.773758 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523685349 usec, L: 0
0627 13:55:41.842836 (+ 69078us) write_op.cc:270] Start()
0627 13:55:41.842867 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523707411 usec, L: 0
0627 13:55:41.844545 (+ 1678us) write_op.cc:270] Start()
0627 13:55:41.844594 (+ 49us) write_op.cc:276] Timestamp: P: 1751032523709471 usec, L: 0
0627 13:55:41.844802 (+ 208us) write_op.cc:270] Start()
0627 13:55:41.844826 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523714436 usec, L: 0
0627 13:55:41.853745 (+ 8919us) write_op.cc:270] Start()
0627 13:55:41.853771 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523742034 usec, L: 0
0627 13:55:41.857198 (+ 3427us) write_op.cc:270] Start()
0627 13:55:41.857222 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523760247 usec, L: 0
0627 13:55:41.865988 (+ 8766us) write_op.cc:270] Start()
0627 13:55:41.866010 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523768654 usec, L: 0
0627 13:55:41.866248 (+ 238us) write_op.cc:270] Start()
0627 13:55:41.866271 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523781338 usec, L: 0
0627 13:55:41.866413 (+ 142us) write_op.cc:270] Start()
0627 13:55:41.866434 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523798642 usec, L: 0
0627 13:55:41.870388 (+ 3954us) write_op.cc:270] Start()
0627 13:55:41.870418 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523811897 usec, L: 0
0627 13:55:41.870625 (+ 207us) write_op.cc:270] Start()
0627 13:55:41.870691 (+ 66us) write_op.cc:276] Timestamp: P: 1751032523822587 usec, L: 0
0627 13:55:41.870873 (+ 182us) write_op.cc:270] Start()
0627 13:55:41.870894 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523845353 usec, L: 0
0627 13:55:41.871067 (+ 173us) write_op.cc:270] Start()
0627 13:55:41.871107 (+ 40us) write_op.cc:276] Timestamp: P: 1751032523856008 usec, L: 0
0627 13:55:41.871274 (+ 167us) write_op.cc:270] Start()
0627 13:55:41.871294 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523859636 usec, L: 0
0627 13:55:41.871485 (+ 191us) write_op.cc:270] Start()
0627 13:55:41.871504 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523864349 usec, L: 0
0627 13:55:41.874371 (+ 2867us) write_op.cc:270] Start()
0627 13:55:41.874400 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523890265 usec, L: 0
0627 13:55:41.875324 (+ 924us) write_op.cc:270] Start()
0627 13:55:41.875350 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523913254 usec, L: 0
0627 13:55:41.875525 (+ 175us) write_op.cc:270] Start()
0627 13:55:41.875546 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523916657 usec, L: 0
0627 13:55:41.894573 (+ 19027us) write_op.cc:270] Start()
0627 13:55:41.894609 (+ 36us) write_op.cc:276] Timestamp: P: 1751032523919140 usec, L: 0
0627 13:55:41.896613 (+ 2004us) write_op.cc:270] Start()
0627 13:55:41.896640 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523947544 usec, L: 0
0627 13:55:41.901530 (+ 4890us) write_op.cc:270] Start()
0627 13:55:41.901558 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523969740 usec, L: 0
0627 13:55:41.904391 (+ 2833us) write_op.cc:270] Start()
0627 13:55:41.904420 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523971092 usec, L: 0
0627 13:55:41.920851 (+ 16431us) write_op.cc:270] Start()
0627 13:55:41.920879 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524000013 usec, L: 0
0627 13:55:41.928728 (+ 7849us) write_op.cc:270] Start()
0627 13:55:41.928761 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524030908 usec, L: 0
0627 13:55:41.928964 (+ 203us) write_op.cc:270] Start()
0627 13:55:41.929013 (+ 49us) write_op.cc:276] Timestamp: P: 1751032524031724 usec, L: 0
0627 13:55:41.929168 (+ 155us) write_op.cc:270] Start()
0627 13:55:41.929191 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524041007 usec, L: 0
0627 13:55:41.932424 (+ 3233us) write_op.cc:270] Start()
0627 13:55:41.932463 (+ 39us) write_op.cc:276] Timestamp: P: 1751032524041695 usec, L: 0
0627 13:55:41.990256 (+ 57793us) write_op.cc:270] Start()
0627 13:55:41.990306 (+ 50us) write_op.cc:276] Timestamp: P: 1751032524088046 usec, L: 0
0627 13:55:41.990560 (+ 254us) write_op.cc:270] Start()
0627 13:55:41.990621 (+ 61us) write_op.cc:276] Timestamp: P: 1751032524088971 usec, L: 0
0627 13:55:41.990784 (+ 163us) write_op.cc:270] Start()
0627 13:55:41.990804 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524089713 usec, L: 0
0627 13:55:41.990950 (+ 146us) write_op.cc:270] Start()
0627 13:55:41.990967 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524116566 usec, L: 0
0627 13:55:41.991157 (+ 190us) write_op.cc:270] Start()
0627 13:55:41.991175 (+ 18us) write_op.cc:276] Timestamp: P: 1751032524143215 usec, L: 0
0627 13:55:42.006774 (+ 15599us) write_op.cc:270] Start()
0627 13:55:42.006805 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524143887 usec, L: 0
0627 13:55:42.019751 (+ 12946us) write_op.cc:270] Start()
0627 13:55:42.019784 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524153382 usec, L: 0
0627 13:55:42.030528 (+ 10744us) write_op.cc:270] Start()
0627 13:55:42.030563 (+ 35us) write_op.cc:276] Timestamp: P: 1751032524165953 usec, L: 0
0627 13:55:42.030765 (+ 202us) write_op.cc:270] Start()
0627 13:55:42.030788 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524184077 usec, L: 0
0627 13:55:42.036327 (+ 5539us) write_op.cc:270] Start()
0627 13:55:42.036358 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524188182 usec, L: 0
0627 13:55:42.049325 (+ 12967us) write_op.cc:270] Start()
0627 13:55:42.049357 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524203398 usec, L: 0
0627 13:55:42.057112 (+ 7755us) write_op.cc:270] Start()
0627 13:55:42.057141 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524223918 usec, L: 0
0627 13:55:42.065266 (+ 8125us) write_op.cc:270] Start()
0627 13:55:42.065309 (+ 43us) write_op.cc:276] Timestamp: P: 1751032524227283 usec, L: 0
0627 13:55:42.069026 (+ 3717us) write_op.cc:270] Start()
0627 13:55:42.069068 (+ 42us) write_op.cc:276] Timestamp: P: 1751032524241845 usec, L: 0
0627 13:55:42.069616 (+ 548us) write_op.cc:270] Start()
0627 13:55:42.069643 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524260820 usec, L: 0
0627 13:55:42.073534 (+ 3891us) write_op.cc:270] Start()
0627 13:55:42.073671 (+ 137us) write_op.cc:276] Timestamp: P: 1751032524279889 usec, L: 0
0627 13:55:42.081282 (+ 7611us) write_op.cc:270] Start()
0627 13:55:42.081316 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524280568 usec, L: 0
0627 13:55:42.081533 (+ 217us) write_op.cc:270] Start()
0627 13:55:42.084569 (+ 3036us) write_op.cc:276] Timestamp: P: 1751032524293603 usec, L: 0
0627 13:55:42.084801 (+ 232us) write_op.cc:270] Start()
0627 13:55:42.084825 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524321145 usec, L: 0
0627 13:55:42.089193 (+ 4368us) write_op.cc:270] Start()
0627 13:55:42.089239 (+ 46us) write_op.cc:276] Timestamp: P: 1751032524322153 usec, L: 0
0627 13:55:42.091225 (+ 1986us) write_op.cc:270] Start()
0627 13:55:42.091252 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524346642 usec, L: 0
0627 13:55:42.094589 (+ 3337us) write_op.cc:270] Start()
0627 13:55:42.094618 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524364605 usec, L: 0
0627 13:55:42.127657 (+ 33039us) write_op.cc:270] Start()
0627 13:55:42.127692 (+ 35us) write_op.cc:276] Timestamp: P: 1751032524376222 usec, L: 0
0627 13:55:42.131941 (+ 4249us) write_op.cc:270] Start()
0627 13:55:42.131979 (+ 38us) write_op.cc:276] Timestamp: P: 1751032524377716 usec, L: 0
0627 13:55:42.132206 (+ 227us) write_op.cc:270] Start()
0627 13:55:42.132231 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524391884 usec, L: 0
0627 13:55:42.133994 (+ 1763us) write_op.cc:270] Start()
0627 13:55:42.134021 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524408557 usec, L: 0
0627 13:55:42.139701 (+ 5680us) write_op.cc:270] Start()
0627 13:55:42.139738 (+ 37us) write_op.cc:276] Timestamp: P: 1751032524427984 usec, L: 0
0627 13:55:42.155536 (+ 15798us) write_op.cc:270] Start()
0627 13:55:42.155568 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524436944 usec, L: 0
0627 13:55:42.162761 (+ 7193us) write_op.cc:270] Start()
0627 13:55:42.162789 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524461222 usec, L: 0
0627 13:55:42.169796 (+ 7007us) write_op.cc:270] Start()
0627 13:55:42.169829 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524463639 usec, L: 0
0627 13:55:42.170079 (+ 250us) write_op.cc:270] Start()
0627 13:55:42.170106 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524477866 usec, L: 0
0627 13:55:42.172178 (+ 2072us) write_op.cc:270] Start()
0627 13:55:42.172203 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524480575 usec, L: 0
0627 13:55:42.174452 (+ 2249us) write_op.cc:270] Start()
0627 13:55:42.174479 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524514456 usec, L: 0
0627 13:55:42.179891 (+ 5412us) write_op.cc:270] Start()
0627 13:55:42.185199 (+ 5308us) write_op.cc:276] Timestamp: P: 1751032524516199 usec, L: 0
0627 13:55:42.195295 (+ 10096us) write_op.cc:270] Start()
0627 13:55:42.195330 (+ 35us) write_op.cc:276] Timestamp: P: 1751032524536713 usec, L: 0
0627 13:55:42.195537 (+ 207us) write_op.cc:270] Start()
0627 13:55:42.195559 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524564125 usec, L: 0
0627 13:55:42.196700 (+ 1141us) write_op.cc:270] Start()
0627 13:55:42.196726 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524579261 usec, L: 0
0627 13:55:42.203111 (+ 6385us) write_op.cc:270] Start()
0627 13:55:42.203142 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524581068 usec, L: 0
0627 13:55:42.208310 (+ 5168us) write_op.cc:270] Start()
0627 13:55:42.208338 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524612440 usec, L: 0
0627 13:55:42.208504 (+ 166us) write_op.cc:270] Start()
0627 13:55:42.208527 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524621536 usec, L: 0
0627 13:55:42.208689 (+ 162us) write_op.cc:270] Start()
0627 13:55:42.208712 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524645789 usec, L: 0
0627 13:55:42.208880 (+ 168us) write_op.cc:270] Start()
0627 13:55:42.208905 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524668182 usec, L: 0
0627 13:55:42.209270 (+ 365us) write_op.cc:270] Start()
0627 13:55:42.209295 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524670065 usec, L: 0
0627 13:55:42.209447 (+ 152us) write_op.cc:270] Start()
0627 13:55:42.209469 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524715189 usec, L: 0
0627 13:55:42.209626 (+ 157us) write_op.cc:270] Start()
0627 13:55:42.209653 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524717005 usec, L: 0
0627 13:55:42.209809 (+ 156us) write_op.cc:270] Start()
0627 13:55:42.209829 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524718045 usec, L: 0
0627 13:55:42.210027 (+ 198us) write_op.cc:270] Start()
0627 13:55:42.210071 (+ 44us) write_op.cc:276] Timestamp: P: 1751032524722407 usec, L: 0
0627 13:55:42.210414 (+ 343us) write_op.cc:270] Start()
0627 13:55:42.210436 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524748285 usec, L: 0
0627 13:55:42.210599 (+ 163us) write_op.cc:270] Start()
0627 13:55:42.210620 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524768278 usec, L: 0
0627 13:55:42.210758 (+ 138us) write_op.cc:270] Start()
0627 13:55:42.210792 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524772543 usec, L: 0
0627 13:55:42.218196 (+ 7404us) write_op.cc:270] Start()
0627 13:55:42.218227 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524773310 usec, L: 0
0627 13:55:42.218447 (+ 220us) write_op.cc:270] Start()
0627 13:55:42.218473 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524802771 usec, L: 0
0627 13:55:42.276235 (+ 57762us) write_op.cc:270] Start()
0627 13:55:42.276263 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524822336 usec, L: 0
0627 13:55:42.276429 (+ 166us) write_op.cc:270] Start()
0627 13:55:42.276452 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524824645 usec, L: 0
0627 13:55:42.276602 (+ 150us) write_op.cc:270] Start()
0627 13:55:42.276643 (+ 41us) write_op.cc:276] Timestamp: P: 1751032524825329 usec, L: 0
0627 13:55:42.276810 (+ 167us) write_op.cc:270] Start()
0627 13:55:42.276842 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524863484 usec, L: 0
0627 13:55:42.277136 (+ 294us) write_op.cc:270] Start()
0627 13:55:42.277160 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524874537 usec, L: 0
0627 13:55:42.295358 (+ 18198us) write_op.cc:270] Start()
0627 13:55:42.295387 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524879925 usec, L: 0
0627 13:55:42.298369 (+ 2982us) write_op.cc:270] Start()
0627 13:55:42.298394 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524880520 usec, L: 0
0627 13:55:42.306604 (+ 8210us) write_op.cc:270] Start()
0627 13:55:42.306628 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524928349 usec, L: 0
0627 13:55:42.329342 (+ 22714us) write_op.cc:270] Start()
0627 13:55:42.329379 (+ 37us) write_op.cc:276] Timestamp: P: 1751032524929463 usec, L: 0
0627 13:55:42.329578 (+ 199us) write_op.cc:270] Start()
0627 13:55:42.329604 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524930142 usec, L: 0
0627 13:55:42.329765 (+ 161us) write_op.cc:270] Start()
0627 13:55:42.329787 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524941131 usec, L: 0
0627 13:55:42.329951 (+ 164us) write_op.cc:270] Start()
0627 13:55:42.329970 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524960972 usec, L: 0
0627 13:55:42.330186 (+ 216us) write_op.cc:270] Start()
0627 13:55:42.330207 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524984235 usec, L: 0
0627 13:55:42.331804 (+ 1597us) write_op.cc:270] Start()
0627 13:55:42.331830 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524986300 usec, L: 0
0627 13:55:42.337800 (+ 5970us) write_op.cc:270] Start()
0627 13:55:42.337830 (+ 30us) write_op.cc:276] Timestamp: P: 1751032525009097 usec, L: 0
0627 13:55:42.338008 (+ 178us) write_op.cc:270] Start()
0627 13:55:42.338066 (+ 58us) write_op.cc:276] Timestamp: P: 1751032525027885 usec, L: 0
0627 13:55:42.360847 (+ 22781us) write_op.cc:270] Start()
0627 13:55:42.360897 (+ 50us) write_op.cc:276] Timestamp: P: 1751032525038820 usec, L: 0
0627 13:55:42.365686 (+ 4789us) write_op.cc:270] Start()
0627 13:55:42.365712 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525040722 usec, L: 0
0627 13:55:42.365916 (+ 204us) write_op.cc:270] Start()
0627 13:55:42.365943 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525069272 usec, L: 0
0627 13:55:42.369895 (+ 3952us) write_op.cc:270] Start()
0627 13:55:42.369922 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525094109 usec, L: 0
0627 13:55:42.370118 (+ 196us) write_op.cc:270] Start()
0627 13:55:42.370139 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525100160 usec, L: 0
0627 13:55:42.370298 (+ 159us) write_op.cc:270] Start()
0627 13:55:42.370319 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525105467 usec, L: 0
0627 13:55:42.370461 (+ 142us) write_op.cc:270] Start()
0627 13:55:42.370479 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525126738 usec, L: 0
0627 13:55:42.370631 (+ 152us) write_op.cc:270] Start()
0627 13:55:42.370656 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525153316 usec, L: 0
0627 13:55:42.371382 (+ 726us) write_op.cc:270] Start()
0627 13:55:42.371408 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525163080 usec, L: 0
0627 13:55:42.371579 (+ 171us) write_op.cc:270] Start()
0627 13:55:42.371599 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525165470 usec, L: 0
0627 13:55:42.371755 (+ 156us) write_op.cc:270] Start()
0627 13:55:42.371775 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525177114 usec, L: 0
0627 13:55:42.371952 (+ 177us) write_op.cc:270] Start()
0627 13:55:42.371973 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525206195 usec, L: 0
0627 13:55:42.372130 (+ 157us) write_op.cc:270] Start()
0627 13:55:42.372150 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525215421 usec, L: 0
0627 13:55:42.372333 (+ 183us) write_op.cc:270] Start()
0627 13:55:42.372356 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525240468 usec, L: 0
0627 13:55:42.372517 (+ 161us) write_op.cc:270] Start()
0627 13:55:42.372544 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525241795 usec, L: 0
0627 13:55:42.372700 (+ 156us) write_op.cc:270] Start()
0627 13:55:42.372718 (+ 18us) write_op.cc:276] Timestamp: P: 1751032525253839 usec, L: 0
0627 13:55:42.372880 (+ 162us) write_op.cc:270] Start()
0627 13:55:42.372900 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525290860 usec, L: 0
0627 13:55:42.373045 (+ 145us) write_op.cc:270] Start()
0627 13:55:42.373064 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525294676 usec, L: 0
0627 13:55:42.373242 (+ 178us) write_op.cc:270] Start()
0627 13:55:42.373262 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525301332 usec, L: 0
0627 13:55:42.373421 (+ 159us) write_op.cc:270] Start()
0627 13:55:42.373441 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525304856 usec, L: 0
0627 13:55:42.373604 (+ 163us) write_op.cc:270] Start()
0627 13:55:42.395156 (+ 21552us) write_op.cc:276] Timestamp: P: 1751032525326604 usec, L: 0
0627 13:55:42.396944 (+ 1788us) write_op.cc:270] Start()
0627 13:55:42.396970 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525352383 usec, L: 0
0627 13:55:42.440259 (+ 43289us) write_op.cc:270] Start()
0627 13:55:42.440285 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525353687 usec, L: 0
0627 13:55:42.440469 (+ 184us) write_op.cc:270] Start()
0627 13:55:42.440490 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525355578 usec, L: 0
0627 13:55:42.454958 (+ 14468us) write_op.cc:270] Start()
0627 13:55:42.454980 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525379672 usec, L: 0
0627 13:55:42.460829 (+ 5849us) write_op.cc:270] Start()
0627 13:55:42.460895 (+ 66us) write_op.cc:276] Timestamp: P: 1751032525391261 usec, L: 0
0627 13:55:42.465286 (+ 4391us) write_op.cc:270] Start()
0627 13:55:42.465315 (+ 29us) write_op.cc:276] Timestamp: P: 1751032525407518 usec, L: 0
0627 13:55:42.496490 (+ 31175us) write_op.cc:270] Start()
0627 13:55:42.496537 (+ 47us) write_op.cc:276] Timestamp: P: 1751032525438920 usec, L: 0
0627 13:55:42.496797 (+ 260us) write_op.cc:270] Start()
0627 13:55:42.496830 (+ 33us) write_op.cc:276] Timestamp: P: 1751032525441835 usec, L: 0
0627 13:55:42.513675 (+ 16845us) write_op.cc:270] Start()
0627 13:55:42.513705 (+ 30us) write_op.cc:276] Timestamp: P: 1751032525447858 usec, L: 0
0627 13:55:42.532541 (+ 18836us) write_op.cc:270] Start()
0627 13:55:42.532568 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525448433 usec, L: 0
0627 13:55:42.532750 (+ 182us) write_op.cc:270] Start()
0627 13:55:42.532776 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525480707 usec, L: 0
0627 13:55:42.532941 (+ 165us) write_op.cc:270] Start()
0627 13:55:42.532963 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525492462 usec, L: 0
0627 13:55:42.533196 (+ 233us) write_op.cc:270] Start()
0627 13:55:42.533219 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525497387 usec, L: 0
0627 13:55:42.533366 (+ 147us) write_op.cc:270] Start()
0627 13:55:42.533383 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525502589 usec, L: 0
0627 13:55:42.533516 (+ 133us) write_op.cc:270] Start()
0627 13:55:42.533536 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525519564 usec, L: 0
0627 13:55:42.533664 (+ 128us) write_op.cc:270] Start()
0627 13:55:42.533681 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525566967 usec, L: 0
0627 13:55:42.533790 (+ 109us) write_op.cc:270] Start()
0627 13:55:42.533806 (+ 16us) write_op.cc:276] Timestamp: P: 1751032525569382 usec, L: 0
0627 13:55:42.533946 (+ 140us) write_op.cc:270] Start()
0627 13:55:42.533965 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525584684 usec, L: 0
0627 13:55:42.534133 (+ 168us) write_op.cc:270] Start()
0627 13:55:42.534152 (+ 19us) write_op.cc:276] Timestamp: P: 1751032525593738 usec, L: 0
0627 13:55:42.534297 (+ 145us) write_op.cc:270] Start()
0627 13:55:42.534314 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525620753 usec, L: 0
0627 13:55:42.534431 (+ 117us) write_op.cc:270] Start()
0627 13:55:42.534446 (+ 15us) write_op.cc:276] Timestamp: P: 1751032525639549 usec, L: 0
0627 13:55:42.534565 (+ 119us) write_op.cc:270] Start()
0627 13:55:42.534582 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525642417 usec, L: 0
0627 13:55:42.534740 (+ 158us) write_op.cc:270] Start()
0627 13:55:42.534765 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525649358 usec, L: 0
0627 13:55:42.534952 (+ 187us) write_op.cc:270] Start()
0627 13:55:42.534969 (+ 17us) write_op.cc:276] Timestamp: P: 1751032525689150 usec, L: 0
0627 13:55:42.564909 (+ 29940us) write_op.cc:270] Start()
0627 13:55:42.564955 (+ 46us) write_op.cc:276] Timestamp: P: 1751032525702040 usec, L: 0
0627 13:55:42.569322 (+ 4367us) write_op.cc:270] Start()
0627 13:55:42.569357 (+ 35us) write_op.cc:276] Timestamp: P: 1751032525713875 usec, L: 0
0627 13:55:42.577618 (+ 8261us) write_op.cc:270] Start()
0627 13:55:42.577649 (+ 31us) write_op.cc:276] Timestamp: P: 1751032525728333 usec, L: 0
0627 13:55:42.591155 (+ 13506us) write_op.cc:270] Start()
0627 13:55:42.591198 (+ 43us) write_op.cc:276] Timestamp: P: 1751032525741194 usec, L: 0
0627 13:55:42.596818 (+ 5620us) write_op.cc:270] Start()
0627 13:55:42.596844 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525764637 usec, L: 0
0627 13:55:42.603484 (+ 6640us) write_op.cc:270] Start()
0627 13:55:42.603515 (+ 31us) write_op.cc:276] Timestamp: P: 1751032525774761 usec, L: 0
0627 13:55:42.605383 (+ 1868us) write_op.cc:270] Start()
0627 13:55:42.605409 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525803162 usec, L: 0
0627 13:55:42.648304 (+ 42895us) write_op.cc:270] Start()
0627 13:55:42.648344 (+ 40us) write_op.cc:276] Timestamp: P: 175103
W20250627 13:55:48.368060 9240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 54255.8us
I20250627 13:55:48.645819 9892 mvcc.cc:204] Tried to move back new op lower bound from 7172229283938283520 to 7172229214043664384. Current Snapshot: MvccSnapshot[applied={T|T < 7172229214110580736}]
I20250627 13:55:49.831669 9945 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:49.833460 9945 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:49.836135 9945 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:49.840803 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1481 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:49.840821 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1481 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:49.842228 9760 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:49.842684 9945 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:55:49.851264 9827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 158) took 1464 ms. Trace:
I20250627 13:55:50.200024 9945 raft_consensus.cc:491] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:50.200201 9945 raft_consensus.cc:513] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } }
I20250627 13:55:50.200802 9945 leader_election.cc:290] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8064817311e242618056800a0ca537dd (127.8.167.195:43925), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:50.201620 9172 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1481 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:50.203259 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ef29c61340f4869b8703e67f2e32f14" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 3 candidate_status { last_received { term: 2 index: 1481 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:50.203840 9760 leader_election.cc:304] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:50.204257 9945 raft_consensus.cc:2747] T 9ef29c61340f4869b8703e67f2e32f14 P 18c6bddc499b4363a95de5d2578e4a07 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:55:49.851388 9827 rpcz_store.cc:276] 0627 13:55:48.385429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:48.385636 (+ 207us) service_pool.cc:225] Handling call
0627 13:55:48.663105 (+277469us) raft_consensus.cc:1469] Updating replica for 426 ops
0627 13:55:48.790537 (+127432us) raft_consensus.cc:1532] Early marking committed up to index 1055
0627 13:55:48.790616 (+ 79us) raft_consensus.cc:1537] Triggering prepare for 426 ops
0627 13:55:48.790724 (+ 108us) write_op.cc:270] Start()
0627 13:55:48.790747 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540065503 usec, L: 0
0627 13:55:48.790910 (+ 163us) write_op.cc:270] Start()
0627 13:55:48.790930 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540095270 usec, L: 0
0627 13:55:48.791071 (+ 141us) write_op.cc:270] Start()
0627 13:55:48.791090 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540096262 usec, L: 0
0627 13:55:48.826452 (+ 35362us) write_op.cc:270] Start()
0627 13:55:48.826482 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540126885 usec, L: 0
0627 13:55:48.829925 (+ 3443us) write_op.cc:270] Start()
0627 13:55:48.829953 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540162088 usec, L: 0
0627 13:55:48.834150 (+ 4197us) write_op.cc:270] Start()
0627 13:55:48.834175 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540162743 usec, L: 0
0627 13:55:48.834398 (+ 223us) write_op.cc:270] Start()
0627 13:55:48.834459 (+ 61us) write_op.cc:276] Timestamp: P: 1751032540165154 usec, L: 0
0627 13:55:48.834651 (+ 192us) write_op.cc:270] Start()
0627 13:55:48.844699 (+ 10048us) write_op.cc:276] Timestamp: P: 1751032540194385 usec, L: 0
0627 13:55:48.856138 (+ 11439us) write_op.cc:270] Start()
0627 13:55:48.856169 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540261430 usec, L: 0
0627 13:55:48.863286 (+ 7117us) write_op.cc:270] Start()
0627 13:55:48.863312 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540263376 usec, L: 0
0627 13:55:48.863491 (+ 179us) write_op.cc:270] Start()
0627 13:55:48.863510 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540272499 usec, L: 0
0627 13:55:48.871410 (+ 7900us) write_op.cc:270] Start()
0627 13:55:48.873109 (+ 1699us) write_op.cc:276] Timestamp: P: 1751032540281811 usec, L: 0
0627 13:55:48.881687 (+ 8578us) write_op.cc:270] Start()
0627 13:55:48.881718 (+ 31us) write_op.cc:276] Timestamp: P: 1751032540286943 usec, L: 0
0627 13:55:48.904918 (+ 23200us) write_op.cc:270] Start()
0627 13:55:48.904946 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540313431 usec, L: 0
0627 13:55:48.905684 (+ 738us) write_op.cc:270] Start()
0627 13:55:48.905710 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540342799 usec, L: 0
0627 13:55:48.905896 (+ 186us) write_op.cc:270] Start()
0627 13:55:48.905918 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540350191 usec, L: 0
0627 13:55:48.907298 (+ 1380us) write_op.cc:270] Start()
0627 13:55:48.907328 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540363131 usec, L: 0
0627 13:55:48.913357 (+ 6029us) write_op.cc:270] Start()
0627 13:55:48.913382 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540381432 usec, L: 0
0627 13:55:48.924497 (+ 11115us) write_op.cc:270] Start()
0627 13:55:48.924535 (+ 38us) write_op.cc:276] Timestamp: P: 1751032540410556 usec, L: 0
0627 13:55:48.938209 (+ 13674us) write_op.cc:270] Start()
0627 13:55:48.938245 (+ 36us) write_op.cc:276] Timestamp: P: 1751032540414180 usec, L: 0
0627 13:55:48.938525 (+ 280us) write_op.cc:270] Start()
0627 13:55:48.938551 (+ 26us) write_op.cc:276] Timestamp: P: 1751032540424345 usec, L: 0
0627 13:55:48.938756 (+ 205us) write_op.cc:270] Start()
0627 13:55:48.938781 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540464423 usec, L: 0
0627 13:55:48.939073 (+ 292us) write_op.cc:270] Start()
0627 13:55:48.939131 (+ 58us) write_op.cc:276] Timestamp: P: 1751032540474517 usec, L: 0
0627 13:55:48.939336 (+ 205us) write_op.cc:270] Start()
0627 13:55:48.939361 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540491717 usec, L: 0
0627 13:55:48.948014 (+ 8653us) write_op.cc:270] Start()
0627 13:55:48.948042 (+ 28us) write_op.cc:276] Timestamp: P: 1751032540492373 usec, L: 0
0627 13:55:48.967611 (+ 19569us) write_op.cc:270] Start()
0627 13:55:48.967641 (+ 30us) write_op.cc:276] Timestamp: P: 1751032540504438 usec, L: 0
0627 13:55:48.968066 (+ 425us) write_op.cc:270] Start()
0627 13:55:48.968091 (+ 25us) write_op.cc:276] Timestamp: P: 1751032540557166 usec, L: 0
0627 13:55:49.002915 (+ 34824us) write_op.cc:270] Start()
0627 13:55:49.002942 (+ 27us) write_op.cc:276] Timestamp: P: 1751032540565730 usec, L: 0
0627 13:55:49.031025 (+ 28083us) write_op.cc:270] Start()
0627 13:55:49.031057 (+ 32us) write_op.cc:276] Timestamp: P: 1751032540575071 usec, L: 0
0627 13:55:49.050228 (+ 19171us) write_op.cc:270] Start()
0627 13:55:49.050264 (+ 36us) write_op.cc:276] Timestamp: P: 1751032540600301 usec, L: 0
0627 13:55:49.050533 (+ 269us) write_op.cc:270] Start()
0627 13:55:49.050555 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540614269 usec, L: 0
0627 13:55:49.050750 (+ 195us) write_op.cc:270] Start()
0627 13:55:49.050770 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540649204 usec, L: 0
0627 13:55:49.050963 (+ 193us) write_op.cc:270] Start()
0627 13:55:49.050984 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540656551 usec, L: 0
0627 13:55:49.051199 (+ 215us) write_op.cc:270] Start()
0627 13:55:49.051221 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540686835 usec, L: 0
0627 13:55:49.051398 (+ 177us) write_op.cc:270] Start()
0627 13:55:49.051422 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540694448 usec, L: 0
0627 13:55:49.051593 (+ 171us) write_op.cc:270] Start()
0627 13:55:49.051613 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540723291 usec, L: 0
0627 13:55:49.051785 (+ 172us) write_op.cc:270] Start()
0627 13:55:49.051803 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540732157 usec, L: 0
0627 13:55:49.051968 (+ 165us) write_op.cc:270] Start()
0627 13:55:49.051986 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540775849 usec, L: 0
0627 13:55:49.052173 (+ 187us) write_op.cc:270] Start()
0627 13:55:49.052192 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540785975 usec, L: 0
0627 13:55:49.052379 (+ 187us) write_op.cc:270] Start()
0627 13:55:49.052402 (+ 23us) write_op.cc:276] Timestamp: P: 1751032540812341 usec, L: 0
0627 13:55:49.052585 (+ 183us) write_op.cc:270] Start()
0627 13:55:49.052604 (+ 19us) write_op.cc:276] Timestamp: P: 1751032540813618 usec, L: 0
0627 13:55:49.052768 (+ 164us) write_op.cc:270] Start()
0627 13:55:49.052790 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540842278 usec, L: 0
0627 13:55:49.052984 (+ 194us) write_op.cc:270] Start()
0627 13:55:49.053005 (+ 21us) write_op.cc:276] Timestamp: P: 1751032540842858 usec, L: 0
0627 13:55:49.053219 (+ 214us) write_op.cc:270] Start()
0627 13:55:49.053239 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540860803 usec, L: 0
0627 13:55:49.053417 (+ 178us) write_op.cc:270] Start()
0627 13:55:49.053439 (+ 22us) write_op.cc:276] Timestamp: P: 1751032540877799 usec, L: 0
0627 13:55:49.053903 (+ 464us) write_op.cc:270] Start()
0627 13:55:49.054177 (+ 274us) write_op.cc:276] Timestamp: P: 1751032540898363 usec, L: 0
0627 13:55:49.097809 (+ 43632us) write_op.cc:270] Start()
0627 13:55:49.097846 (+ 37us) write_op.cc:276] Timestamp: P: 1751032540927840 usec, L: 0
0627 13:55:49.109332 (+ 11486us) write_op.cc:270] Start()
0627 13:55:49.109356 (+ 24us) write_op.cc:276] Timestamp: P: 1751032540942544 usec, L: 0
0627 13:55:49.109598 (+ 242us) write_op.cc:270] Start()
0627 13:55:49.109618 (+ 20us) write_op.cc:276] Timestamp: P: 1751032540948961 usec, L: 0
0627 13:55:49.109799 (+ 181us) write_op.cc:270] Start()
0627 13:55:49.109817 (+ 18us) write_op.cc:276] Timestamp: P: 1751032540954589 usec, L: 0
0627 13:55:49.109981 (+ 164us) write_op.cc:270] Start()
0627 13:55:49.109997 (+ 16us) write_op.cc:276] Timestamp: P: 1751032540994095 usec, L: 0
0627 13:55:49.117664 (+ 7667us) write_op.cc:270] Start()
0627 13:55:49.117688 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541008816 usec, L: 0
0627 13:55:49.117884 (+ 196us) write_op.cc:270] Start()
0627 13:55:49.117906 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541011860 usec, L: 0
0627 13:55:49.118120 (+ 214us) write_op.cc:270] Start()
0627 13:55:49.118142 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541032276 usec, L: 0
0627 13:55:49.142254 (+ 24112us) write_op.cc:270] Start()
0627 13:55:49.142281 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541061969 usec, L: 0
0627 13:55:49.156627 (+ 14346us) write_op.cc:270] Start()
0627 13:55:49.156653 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541067628 usec, L: 0
0627 13:55:49.156845 (+ 192us) write_op.cc:270] Start()
0627 13:55:49.156866 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541088174 usec, L: 0
0627 13:55:49.157025 (+ 159us) write_op.cc:270] Start()
0627 13:55:49.157045 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541110482 usec, L: 0
0627 13:55:49.157209 (+ 164us) write_op.cc:270] Start()
0627 13:55:49.157232 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541152891 usec, L: 0
0627 13:55:49.157384 (+ 152us) write_op.cc:270] Start()
0627 13:55:49.158018 (+ 634us) write_op.cc:276] Timestamp: P: 1751032541160552 usec, L: 0
0627 13:55:49.158250 (+ 232us) write_op.cc:270] Start()
0627 13:55:49.158275 (+ 25us) write_op.cc:276] Timestamp: P: 1751032541171744 usec, L: 0
0627 13:55:49.158435 (+ 160us) write_op.cc:270] Start()
0627 13:55:49.181729 (+ 23294us) write_op.cc:276] Timestamp: P: 1751032541190233 usec, L: 0
0627 13:55:49.198318 (+ 16589us) write_op.cc:270] Start()
0627 13:55:49.198346 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541226693 usec, L: 0
0627 13:55:49.198585 (+ 239us) write_op.cc:270] Start()
0627 13:55:49.198609 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541227277 usec, L: 0
0627 13:55:49.198820 (+ 211us) write_op.cc:270] Start()
0627 13:55:49.198843 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541257248 usec, L: 0
0627 13:55:49.210416 (+ 11573us) write_op.cc:270] Start()
0627 13:55:49.210442 (+ 26us) write_op.cc:276] Timestamp: P: 1751032541281898 usec, L: 0
0627 13:55:49.210602 (+ 160us) write_op.cc:270] Start()
0627 13:55:49.210620 (+ 18us) write_op.cc:276] Timestamp: P: 1751032541298674 usec, L: 0
0627 13:55:49.210779 (+ 159us) write_op.cc:270] Start()
0627 13:55:49.210798 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541314206 usec, L: 0
0627 13:55:49.210973 (+ 175us) write_op.cc:270] Start()
0627 13:55:49.210995 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541318353 usec, L: 0
0627 13:55:49.211218 (+ 223us) write_op.cc:270] Start()
0627 13:55:49.211241 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541335052 usec, L: 0
0627 13:55:49.211509 (+ 268us) write_op.cc:270] Start()
0627 13:55:49.211532 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541363585 usec, L: 0
0627 13:55:49.211719 (+ 187us) write_op.cc:270] Start()
0627 13:55:49.211741 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541406484 usec, L: 0
0627 13:55:49.211925 (+ 184us) write_op.cc:270] Start()
0627 13:55:49.211955 (+ 30us) write_op.cc:276] Timestamp: P: 1751032541416833 usec, L: 0
0627 13:55:49.212188 (+ 233us) write_op.cc:270] Start()
0627 13:55:49.212208 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541420151 usec, L: 0
0627 13:55:49.212390 (+ 182us) write_op.cc:270] Start()
0627 13:55:49.212414 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541449507 usec, L: 0
0627 13:55:49.212597 (+ 183us) write_op.cc:270] Start()
0627 13:55:49.212639 (+ 42us) write_op.cc:276] Timestamp: P: 1751032541482396 usec, L: 0
0627 13:55:49.212830 (+ 191us) write_op.cc:270] Start()
0627 13:55:49.212851 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541490385 usec, L: 0
0627 13:55:49.219506 (+ 6655us) write_op.cc:270] Start()
0627 13:55:49.219529 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541497027 usec, L: 0
0627 13:55:49.282233 (+ 62704us) write_op.cc:270] Start()
0627 13:55:49.282257 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541508726 usec, L: 0
0627 13:55:49.282460 (+ 203us) write_op.cc:270] Start()
0627 13:55:49.282483 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541549187 usec, L: 0
0627 13:55:49.282689 (+ 206us) write_op.cc:270] Start()
0627 13:55:49.282709 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541577880 usec, L: 0
0627 13:55:49.282857 (+ 148us) write_op.cc:270] Start()
0627 13:55:49.282877 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541583785 usec, L: 0
0627 13:55:49.283009 (+ 132us) write_op.cc:270] Start()
0627 13:55:49.283028 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541592793 usec, L: 0
0627 13:55:49.283203 (+ 175us) write_op.cc:270] Start()
0627 13:55:49.283224 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541619445 usec, L: 0
0627 13:55:49.283405 (+ 181us) write_op.cc:270] Start()
0627 13:55:49.283434 (+ 29us) write_op.cc:276] Timestamp: P: 1751032541647021 usec, L: 0
0627 13:55:49.283602 (+ 168us) write_op.cc:270] Start()
0627 13:55:49.283621 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541649532 usec, L: 0
0627 13:55:49.283775 (+ 154us) write_op.cc:270] Start()
0627 13:55:49.283799 (+ 24us) write_op.cc:276] Timestamp: P: 1751032541678485 usec, L: 0
0627 13:55:49.283939 (+ 140us) write_op.cc:270] Start()
0627 13:55:49.283959 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541718962 usec, L: 0
0627 13:55:49.284129 (+ 170us) write_op.cc:270] Start()
0627 13:55:49.284149 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541719590 usec, L: 0
0627 13:55:49.284338 (+ 189us) write_op.cc:270] Start()
0627 13:55:49.284357 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541721497 usec, L: 0
0627 13:55:49.284501 (+ 144us) write_op.cc:270] Start()
0627 13:55:49.284521 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541760797 usec, L: 0
0627 13:55:49.284669 (+ 148us) write_op.cc:270] Start()
0627 13:55:49.284689 (+ 20us) write_op.cc:276] Timestamp: P: 1751032541794557 usec, L: 0
0627 13:55:49.284820 (+ 131us) write_op.cc:270] Start()
0627 13:55:49.284839 (+ 19us) write_op.cc:276] Timestamp: P: 1751032541795900 usec, L: 0
0627 13:55:49.292369 (+ 7530us) write_op.cc:270] Start()
0627 13:55:49.292397 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541814982 usec, L: 0
0627 13:55:49.292605 (+ 208us) write_op.cc:270] Start()
0627 13:55:49.292633 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541848706 usec, L: 0
0627 13:55:49.292836 (+ 203us) write_op.cc:270] Start()
0627 13:55:49.292864 (+ 28us) write_op.cc:276] Timestamp: P: 1751032541885945 usec, L: 0
0627 13:55:49.293062 (+ 198us) write_op.cc:270] Start()
0627 13:55:49.293084 (+ 22us) write_op.cc:276] Timestamp: P: 1751032541886997 usec, L: 0
0627 13:55:49.293266 (+ 182us) write_op.cc:270] Start()
0627 13:55:49.293289 (+ 23us) write_op.cc:276] Timestamp: P: 1751032541896556 usec, L: 0
0627 13:55:49.293456 (+ 167us) write_op.cc:270] Start()
0627 13:55:49.293477 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541920451 usec, L: 0
0627 13:55:49.293623 (+ 146us) write_op.cc:270] Start()
0627 13:55:49.293650 (+ 27us) write_op.cc:276] Timestamp: P: 1751032541945239 usec, L: 0
0627 13:55:49.293804 (+ 154us) write_op.cc:270] Start()
0627 13:55:49.293825 (+ 21us) write_op.cc:276] Timestamp: P: 1751032541947504 usec, L: 0
0627 13:55:49.294025 (+ 200us) write_op.cc:270] Start()
0627 13:55:49.294070 (+ 45us) write_op.cc:276] Timestamp: P: 1751032541962482 usec, L: 0
0627 13:55:49.294918 (+ 848us) write_op.cc:270] Start()
0627 13:55:49.294944 (+ 26us) write_op.cc:276] Timestamp: P: 1751032542012365 usec, L: 0
0627 13:55:49.302259 (+ 7315us) write_op.cc:270] Start()
0627 13:55:49.302287 (+ 28us) write_op.cc:276] Timestamp: P: 1751032542072046 usec, L: 0
0627 13:55:49.307649 (+ 5362us) write_op.cc:270] Start()
0627 13:55:49.307674 (+ 25us) write_op.cc:276] Timestamp: P: 1751032542083422 usec, L: 0
0627 13:55:49.318651 (+ 10977us) write_op.cc:270] Start()
0627 13:55:49.318678 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542098959 usec, L: 0
0627 13:55:49.318883 (+ 205us) write_op.cc:270] Start()
0627 13:55:49.318905 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542118007 usec, L: 0
0627 13:55:49.319129 (+ 224us) write_op.cc:270] Start()
0627 13:55:49.319170 (+ 41us) write_op.cc:276] Timestamp: P: 1751032542119615 usec, L: 0
0627 13:55:49.319336 (+ 166us) write_op.cc:270] Start()
0627 13:55:49.319360 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542129105 usec, L: 0
0627 13:55:49.319562 (+ 202us) write_op.cc:270] Start()
0627 13:55:49.319581 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542170931 usec, L: 0
0627 13:55:49.319729 (+ 148us) write_op.cc:270] Start()
0627 13:55:49.319750 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542176129 usec, L: 0
0627 13:55:49.319938 (+ 188us) write_op.cc:270] Start()
0627 13:55:49.319958 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542177430 usec, L: 0
0627 13:55:49.320237 (+ 279us) write_op.cc:270] Start()
0627 13:55:49.320257 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542205870 usec, L: 0
0627 13:55:49.320483 (+ 226us) write_op.cc:270] Start()
0627 13:55:49.320505 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542235471 usec, L: 0
0627 13:55:49.320676 (+ 171us) write_op.cc:270] Start()
0627 13:55:49.320696 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542237837 usec, L: 0
0627 13:55:49.320872 (+ 176us) write_op.cc:270] Start()
0627 13:55:49.320892 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542246168 usec, L: 0
0627 13:55:49.321062 (+ 170us) write_op.cc:270] Start()
0627 13:55:49.321082 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542265790 usec, L: 0
0627 13:55:49.321263 (+ 181us) write_op.cc:270] Start()
0627 13:55:49.321283 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542320844 usec, L: 0
0627 13:55:49.321478 (+ 195us) write_op.cc:270] Start()
0627 13:55:49.321499 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542332670 usec, L: 0
0627 13:55:49.321672 (+ 173us) write_op.cc:270] Start()
0627 13:55:49.321692 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542342978 usec, L: 0
0627 13:55:49.321862 (+ 170us) write_op.cc:270] Start()
0627 13:55:49.321883 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542354960 usec, L: 0
0627 13:55:49.322111 (+ 228us) write_op.cc:270] Start()
0627 13:55:49.322134 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542407225 usec, L: 0
0627 13:55:49.322335 (+ 201us) write_op.cc:270] Start()
0627 13:55:49.322356 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542413294 usec, L: 0
0627 13:55:49.322546 (+ 190us) write_op.cc:270] Start()
0627 13:55:49.322567 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542430581 usec, L: 0
0627 13:55:49.323217 (+ 650us) write_op.cc:270] Start()
0627 13:55:49.323241 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542435959 usec, L: 0
0627 13:55:49.323413 (+ 172us) write_op.cc:270] Start()
0627 13:55:49.323434 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542446755 usec, L: 0
0627 13:55:49.330200 (+ 6766us) write_op.cc:270] Start()
0627 13:55:49.330222 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542451478 usec, L: 0
0627 13:55:49.330603 (+ 381us) write_op.cc:270] Start()
0627 13:55:49.330624 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542482527 usec, L: 0
0627 13:55:49.331047 (+ 423us) write_op.cc:270] Start()
0627 13:55:49.331067 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542543099 usec, L: 0
0627 13:55:49.332278 (+ 1211us) write_op.cc:270] Start()
0627 13:55:49.332302 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542547210 usec, L: 0
0627 13:55:49.333195 (+ 893us) write_op.cc:270] Start()
0627 13:55:49.333241 (+ 46us) write_op.cc:276] Timestamp: P: 1751032542569731 usec, L: 0
0627 13:55:49.337323 (+ 4082us) write_op.cc:270] Start()
0627 13:55:49.337345 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542578195 usec, L: 0
0627 13:55:49.344423 (+ 7078us) write_op.cc:270] Start()
0627 13:55:49.344446 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542586462 usec, L: 0
0627 13:55:49.344671 (+ 225us) write_op.cc:270] Start()
0627 13:55:49.344693 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542628689 usec, L: 0
0627 13:55:49.361282 (+ 16589us) write_op.cc:270] Start()
0627 13:55:49.361312 (+ 30us) write_op.cc:276] Timestamp: P: 1751032542650337 usec, L: 0
0627 13:55:49.363182 (+ 1870us) write_op.cc:270] Start()
0627 13:55:49.363209 (+ 27us) write_op.cc:276] Timestamp: P: 1751032542653901 usec, L: 0
0627 13:55:49.363426 (+ 217us) write_op.cc:270] Start()
0627 13:55:49.363450 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542685478 usec, L: 0
0627 13:55:49.364340 (+ 890us) write_op.cc:270] Start()
0627 13:55:49.364366 (+ 26us) write_op.cc:276] Timestamp: P: 1751032542699464 usec, L: 0
0627 13:55:49.366016 (+ 1650us) write_op.cc:270] Start()
0627 13:55:49.383662 (+ 17646us) write_op.cc:276] Timestamp: P: 1751032542743137 usec, L: 0
0627 13:55:49.383908 (+ 246us) write_op.cc:270] Start()
0627 13:55:49.383934 (+ 26us) write_op.cc:276] Timestamp: P: 1751032542769792 usec, L: 0
0627 13:55:49.384116 (+ 182us) write_op.cc:270] Start()
0627 13:55:49.384137 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542770718 usec, L: 0
0627 13:55:49.384322 (+ 185us) write_op.cc:270] Start()
0627 13:55:49.384343 (+ 21us) write_op.cc:276] Timestamp: P: 1751032542772891 usec, L: 0
0627 13:55:49.384502 (+ 159us) write_op.cc:270] Start()
0627 13:55:49.384522 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542789593 usec, L: 0
0627 13:55:49.388541 (+ 4019us) write_op.cc:270] Start()
0627 13:55:49.388563 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542812122 usec, L: 0
0627 13:55:49.388744 (+ 181us) write_op.cc:270] Start()
0627 13:55:49.388767 (+ 23us) write_op.cc:276] Timestamp: P: 1751032542812654 usec, L: 0
0627 13:55:49.388953 (+ 186us) write_op.cc:270] Start()
0627 13:55:49.388977 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542847610 usec, L: 0
0627 13:55:49.389521 (+ 544us) write_op.cc:270] Start()
0627 13:55:49.389545 (+ 24us) write_op.cc:276] Timestamp: P: 1751032542894660 usec, L: 0
0627 13:55:49.389695 (+ 150us) write_op.cc:270] Start()
0627 13:55:49.389715 (+ 20us) write_op.cc:276] Timestamp: P: 1751032542904675 usec, L: 0
0627 13:55:49.389846 (+ 131us) write_op.cc:270] Start()
0627 13:55:49.389868 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542905259 usec, L: 0
0627 13:55:49.389994 (+ 126us) write_op.cc:270] Start()
0627 13:55:49.390013 (+ 19us) write_op.cc:276] Timestamp: P: 1751032542915539 usec, L: 0
0627 13:55:49.402206 (+ 12193us) write_op.cc:270] Start()
0627 13:55:49.402241 (+ 35us) write_op.cc:276] Timestamp: P: 1751032542948379 usec, L: 0
0627 13:55:49.402429 (+ 188us) write_op.cc:270] Start()
0627 13:55:49.402451 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542969007 usec, L: 0
0627 13:55:49.402619 (+ 168us) write_op.cc:270] Start()
0627 13:55:49.402641 (+ 22us) write_op.cc:276] Timestamp: P: 1751032542978240 usec, L: 0
0627 13:55:49.402811 (+ 170us) write_op.cc:270] Start()
0627 13:55:49.402832 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543006963 usec, L: 0
0627 13:55:49.402999 (+ 167us) write_op.cc:270] Start()
0627 13:55:49.403020 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543017853 usec, L: 0
0627 13:55:49.403195 (+ 175us) write_op.cc:270] Start()
0627 13:55:49.403226 (+ 31us) write_op.cc:276] Timestamp: P: 1751032543034975 usec, L: 0
0627 13:55:49.403414 (+ 188us) write_op.cc:270] Start()
0627 13:55:49.403438 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543076079 usec, L: 0
0627 13:55:49.403627 (+ 189us) write_op.cc:270] Start()
0627 13:55:49.403652 (+ 25us) write_op.cc:276] Timestamp: P: 1751032543088463 usec, L: 0
0627 13:55:49.424695 (+ 21043us) write_op.cc:270] Start()
0627 13:55:49.424718 (+ 23us) write_op.cc:276] Timestamp: P: 1751032543115904 usec, L: 0
0627 13:55:49.425228 (+ 510us) write_op.cc:270] Start()
0627 13:55:49.425253 (+ 25us) write_op.cc:276] Timestamp: P: 1751032543138768 usec, L: 0
0627 13:55:49.425660 (+ 407us) write_op.cc:270] Start()
0627 13:55:49.425681 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543167635 usec, L: 0
0627 13:55:49.426118 (+ 437us) write_op.cc:270] Start()
0627 13:55:49.426138 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543169067 usec, L: 0
0627 13:55:49.426721 (+ 583us) write_op.cc:270] Start()
0627 13:55:49.426745 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543196366 usec, L: 0
0627 13:55:49.427157 (+ 412us) write_op.cc:270] Start()
0627 13:55:49.427178 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543206282 usec, L: 0
0627 13:55:49.427592 (+ 414us) write_op.cc:270] Start()
0627 13:55:49.427614 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543223072 usec, L: 0
0627 13:55:49.428036 (+ 422us) write_op.cc:270] Start()
0627 13:55:49.428057 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543229453 usec, L: 0
0627 13:55:49.428460 (+ 403us) write_op.cc:270] Start()
0627 13:55:49.428483 (+ 23us) write_op.cc:276] Timestamp: P: 1751032543260554 usec, L: 0
0627 13:55:49.428890 (+ 407us) write_op.cc:270] Start()
0627 13:55:49.428910 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543268403 usec, L: 0
0627 13:55:49.429324 (+ 414us) write_op.cc:270] Start()
0627 13:55:49.429346 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543287516 usec, L: 0
0627 13:55:49.429693 (+ 347us) write_op.cc:270] Start()
0627 13:55:49.429714 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543293840 usec, L: 0
0627 13:55:49.430510 (+ 796us) write_op.cc:270] Start()
0627 13:55:49.430534 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543336778 usec, L: 0
0627 13:55:49.441132 (+ 10598us) write_op.cc:270] Start()
0627 13:55:49.441157 (+ 25us) write_op.cc:276] Timestamp: P: 1751032543340167 usec, L: 0
0627 13:55:49.441618 (+ 461us) write_op.cc:270] Start()
0627 13:55:49.441643 (+ 25us) write_op.cc:276] Timestamp: P: 1751032543348535 usec, L: 0
0627 13:55:49.441832 (+ 189us) write_op.cc:270] Start()
0627 13:55:49.441852 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543349841 usec, L: 0
0627 13:55:49.442026 (+ 174us) write_op.cc:270] Start()
0627 13:55:49.442068 (+ 42us) write_op.cc:276] Timestamp: P: 1751032543375883 usec, L: 0
0627 13:55:49.442227 (+ 159us) write_op.cc:270] Start()
0627 13:55:49.442250 (+ 23us) write_op.cc:276] Timestamp: P: 1751032543400601 usec, L: 0
0627 13:55:49.442418 (+ 168us) write_op.cc:270] Start()
0627 13:55:49.442445 (+ 27us) write_op.cc:276] Timestamp: P: 1751032543412027 usec, L: 0
0627 13:55:49.442659 (+ 214us) write_op.cc:270] Start()
0627 13:55:49.442680 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543421205 usec, L: 0
0627 13:55:49.442849 (+ 169us) write_op.cc:270] Start()
0627 13:55:49.442868 (+ 19us) write_op.cc:276] Timestamp: P: 1751032543440989 usec, L: 0
0627 13:55:49.443029 (+ 161us) write_op.cc:270] Start()
0627 13:55:49.443050 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543472590 usec, L: 0
0627 13:55:49.443228 (+ 178us) write_op.cc:270] Start()
0627 13:55:49.443251 (+ 23us) write_op.cc:276] Timestamp: P: 1751032543491327 usec, L: 0
0627 13:55:49.443421 (+ 170us) write_op.cc:270] Start()
0627 13:55:49.443443 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543500210 usec, L: 0
0627 13:55:49.443612 (+ 169us) write_op.cc:270] Start()
0627 13:55:49.443633 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543524614 usec, L: 0
0627 13:55:49.443798 (+ 165us) write_op.cc:270] Start()
0627 13:55:49.443819 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543550927 usec, L: 0
0627 13:55:49.447856 (+ 4037us) write_op.cc:270] Start()
0627 13:55:49.447880 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543579468 usec, L: 0
0627 13:55:49.448125 (+ 245us) write_op.cc:270] Start()
0627 13:55:49.448146 (+ 21us) write_op.cc:276] Timestamp: P: 1751032543611049 usec, L: 0
0627 13:55:49.448345 (+ 199us) write_op.cc:270] Start()
0627 13:55:49.448361 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543612008 usec, L: 0
0627 13:55:49.448535 (+ 174us) write_op.cc:270] Start()
0627 13:55:49.448551 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543613227 usec, L: 0
0627 13:55:49.448741 (+ 190us) write_op.cc:270] Start()
0627 13:55:49.448763 (+ 22us) write_op.cc:276] Timestamp: P: 1751032543659115 usec, L: 0
0627 13:55:49.448932 (+ 169us) write_op.cc:270] Start()
0627 13:55:49.448952 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543694928 usec, L: 0
0627 13:55:49.449124 (+ 172us) write_op.cc:270] Start()
0627 13:55:49.449144 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543699858 usec, L: 0
0627 13:55:49.449350 (+ 206us) write_op.cc:270] Start()
0627 13:55:49.449370 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543704218 usec, L: 0
0627 13:55:49.449570 (+ 200us) write_op.cc:270] Start()
0627 13:55:49.449590 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543719559 usec, L: 0
0627 13:55:49.449803 (+ 213us) write_op.cc:270] Start()
0627 13:55:49.449819 (+ 16us) write_op.cc:276] Timestamp: P: 1751032543776052 usec, L: 0
0627 13:55:49.450072 (+ 253us) write_op.cc:270] Start()
0627 13:55:49.450096 (+ 24us) write_op.cc:276] Timestamp: P: 1751032543777126 usec, L: 0
0627 13:55:49.450303 (+ 207us) write_op.cc:270] Start()
0627 13:55:49.450323 (+ 20us) write_op.cc:276] Timestamp: P: 1751032543777787 usec, L: 0
0627 13:55:49.450505 (+ 182us) write_op.cc:270] Start()
0627 13:55:49.450529 (+ 24us) write_op.cc:276] Timestamp: P: 17510
I20250627 13:55:50.838598 9830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.8.167.195:59563 (request call id 90) took 4280 ms. Trace:
I20250627 13:55:50.839042 9787 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:57736 (request call id 54) took 3132 ms. Trace:
I20250627 13:55:50.839272 9920 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:50.843398 9920 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:50.843336 9787 rpcz_store.cc:276] 0627 13:55:47.706311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:47.706560 (+ 249us) service_pool.cc:225] Handling call
0627 13:55:50.838413 (+3131853us) spinlock_profiling.cc:243] Waited 3.13 s on lock 0x616000019354. stack: 00007f2433513316 00007f24335130d0 00007f24325093ad 0000000000971004 0000000000970fcd 000000000097099a 00007f243ef43100 00007f2442cd65af 00007f243d415a36 00007f243d415765 00007f243a2b6c7e 00007f243a2b5b5e 00007f243a2bbd95 00007f243a2bc6bf 00007f243a2bc4e8 0000000000a7573c
0627 13:55:50.839019 (+ 606us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7200185600}
I20250627 13:55:50.844408 9920 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:50.853876 9302 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1383 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:50.854758 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1383 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:50.857460 9761 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:50.858907 9920 raft_consensus.cc:2747] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:50.866320 9109 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.195:43925, user_credentials={real_user=slave}} blocked reactor thread for 50486.9us
W20250627 13:55:51.029735 9109 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.193:45719, user_credentials={real_user=slave}} blocked reactor thread for 61284.6us
I20250627 13:55:51.231989 9920 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:51.232157 9920 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:51.232784 9920 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:51.236236 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1383 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:51.237223 9301 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1383 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:51.237733 9760 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:51.239199 9920 raft_consensus.cc:2747] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:55:50.844766 9830 rpcz_store.cc:276] 0627 13:55:46.558267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0627 13:55:46.558633 (+ 366us) service_pool.cc:225] Handling call
0627 13:55:47.389126 (+830493us) raft_consensus.cc:1469] Updating replica for 1227 ops
0627 13:55:47.845682 (+456556us) raft_consensus.cc:1532] Early marking committed up to index 158
0627 13:55:47.845823 (+ 141us) raft_consensus.cc:1537] Triggering prepare for 1225 ops
0627 13:55:47.846005 (+ 182us) write_op.cc:270] Start()
0627 13:55:47.846064 (+ 59us) write_op.cc:276] Timestamp: P: 1751032522420843 usec, L: 0
0627 13:55:47.846316 (+ 252us) write_op.cc:270] Start()
0627 13:55:47.846344 (+ 28us) write_op.cc:276] Timestamp: P: 1751032522450912 usec, L: 0
0627 13:55:47.846561 (+ 217us) write_op.cc:270] Start()
0627 13:55:47.846584 (+ 23us) write_op.cc:276] Timestamp: P: 1751032522653921 usec, L: 0
0627 13:55:47.846814 (+ 230us) write_op.cc:270] Start()
0627 13:55:47.846861 (+ 47us) write_op.cc:276] Timestamp: P: 1751032522666154 usec, L: 0
0627 13:55:47.857466 (+ 10605us) write_op.cc:270] Start()
0627 13:55:47.857497 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523000473 usec, L: 0
0627 13:55:47.858264 (+ 767us) write_op.cc:270] Start()
0627 13:55:47.858287 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523039212 usec, L: 0
0627 13:55:47.858489 (+ 202us) write_op.cc:270] Start()
0627 13:55:47.858557 (+ 68us) write_op.cc:276] Timestamp: P: 1751032523041625 usec, L: 0
0627 13:55:47.858756 (+ 199us) write_op.cc:270] Start()
0627 13:55:47.858780 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523042971 usec, L: 0
0627 13:55:47.858997 (+ 217us) write_op.cc:270] Start()
0627 13:55:47.859018 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523074414 usec, L: 0
0627 13:55:47.859239 (+ 221us) write_op.cc:270] Start()
0627 13:55:47.859263 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523075431 usec, L: 0
0627 13:55:47.859477 (+ 214us) write_op.cc:270] Start()
0627 13:55:47.859500 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523120999 usec, L: 0
0627 13:55:47.859692 (+ 192us) write_op.cc:270] Start()
0627 13:55:47.859712 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523123180 usec, L: 0
0627 13:55:47.859910 (+ 198us) write_op.cc:270] Start()
0627 13:55:47.859941 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523134674 usec, L: 0
0627 13:55:47.860135 (+ 194us) write_op.cc:270] Start()
0627 13:55:47.860159 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523137446 usec, L: 0
0627 13:55:47.860376 (+ 217us) write_op.cc:270] Start()
0627 13:55:47.860409 (+ 33us) write_op.cc:276] Timestamp: P: 1751032523153270 usec, L: 0
0627 13:55:47.860620 (+ 211us) write_op.cc:270] Start()
0627 13:55:47.860640 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523185406 usec, L: 0
0627 13:55:47.860904 (+ 264us) write_op.cc:270] Start()
0627 13:55:47.860937 (+ 33us) write_op.cc:276] Timestamp: P: 1751032523187044 usec, L: 0
0627 13:55:47.861135 (+ 198us) write_op.cc:270] Start()
0627 13:55:47.861157 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523189682 usec, L: 0
0627 13:55:47.861361 (+ 204us) write_op.cc:270] Start()
0627 13:55:47.861385 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523208470 usec, L: 0
0627 13:55:47.861572 (+ 187us) write_op.cc:270] Start()
0627 13:55:47.861607 (+ 35us) write_op.cc:276] Timestamp: P: 1751032523227167 usec, L: 0
0627 13:55:47.861841 (+ 234us) write_op.cc:270] Start()
0627 13:55:47.861871 (+ 30us) write_op.cc:276] Timestamp: P: 1751032523239410 usec, L: 0
0627 13:55:47.862151 (+ 280us) write_op.cc:270] Start()
0627 13:55:47.862174 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523248816 usec, L: 0
0627 13:55:47.862387 (+ 213us) write_op.cc:270] Start()
0627 13:55:47.862409 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523259755 usec, L: 0
0627 13:55:47.870354 (+ 7945us) write_op.cc:270] Start()
0627 13:55:47.870382 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523287993 usec, L: 0
0627 13:55:47.871188 (+ 806us) write_op.cc:270] Start()
0627 13:55:47.871219 (+ 31us) write_op.cc:276] Timestamp: P: 1751032523308929 usec, L: 0
0627 13:55:47.871418 (+ 199us) write_op.cc:270] Start()
0627 13:55:47.871443 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523312925 usec, L: 0
0627 13:55:47.878398 (+ 6955us) write_op.cc:270] Start()
0627 13:55:47.878423 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523318926 usec, L: 0
0627 13:55:47.878606 (+ 183us) write_op.cc:270] Start()
0627 13:55:47.878627 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523330781 usec, L: 0
0627 13:55:47.880148 (+ 1521us) write_op.cc:270] Start()
0627 13:55:47.880166 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523345551 usec, L: 0
0627 13:55:47.880350 (+ 184us) write_op.cc:270] Start()
0627 13:55:47.880371 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523376438 usec, L: 0
0627 13:55:47.880536 (+ 165us) write_op.cc:270] Start()
0627 13:55:47.880563 (+ 27us) write_op.cc:276] Timestamp: P: 1751032523390668 usec, L: 0
0627 13:55:47.880791 (+ 228us) write_op.cc:270] Start()
0627 13:55:47.880812 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523392670 usec, L: 0
0627 13:55:47.880983 (+ 171us) write_op.cc:270] Start()
0627 13:55:47.881005 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523398158 usec, L: 0
0627 13:55:47.881212 (+ 207us) write_op.cc:270] Start()
0627 13:55:47.881233 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523427038 usec, L: 0
0627 13:55:47.881419 (+ 186us) write_op.cc:270] Start()
0627 13:55:47.881462 (+ 43us) write_op.cc:276] Timestamp: P: 1751032523439717 usec, L: 0
0627 13:55:47.881749 (+ 287us) write_op.cc:270] Start()
0627 13:55:47.881771 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523453932 usec, L: 0
0627 13:55:47.881984 (+ 213us) write_op.cc:270] Start()
0627 13:55:47.882004 (+ 20us) write_op.cc:276] Timestamp: P: 1751032523456962 usec, L: 0
0627 13:55:47.882194 (+ 190us) write_op.cc:270] Start()
0627 13:55:47.882210 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523462829 usec, L: 0
0627 13:55:47.882715 (+ 505us) write_op.cc:270] Start()
0627 13:55:47.882740 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523499978 usec, L: 0
0627 13:55:47.883864 (+ 1124us) write_op.cc:270] Start()
0627 13:55:47.883889 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523508164 usec, L: 0
0627 13:55:47.884073 (+ 184us) write_op.cc:270] Start()
0627 13:55:47.884094 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523524850 usec, L: 0
0627 13:55:47.886565 (+ 2471us) write_op.cc:270] Start()
0627 13:55:47.886590 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523537216 usec, L: 0
0627 13:55:47.889009 (+ 2419us) write_op.cc:270] Start()
0627 13:55:47.889037 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523537851 usec, L: 0
0627 13:55:47.889218 (+ 181us) write_op.cc:270] Start()
0627 13:55:47.889246 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523571266 usec, L: 0
0627 13:55:47.889800 (+ 554us) write_op.cc:270] Start()
0627 13:55:47.889824 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523583238 usec, L: 0
0627 13:55:47.890092 (+ 268us) write_op.cc:270] Start()
0627 13:55:47.890116 (+ 24us) write_op.cc:276] Timestamp: P: 1751032523597140 usec, L: 0
0627 13:55:47.891457 (+ 1341us) write_op.cc:270] Start()
0627 13:55:47.891480 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523602214 usec, L: 0
0627 13:55:47.891684 (+ 204us) write_op.cc:270] Start()
0627 13:55:47.891707 (+ 23us) write_op.cc:276] Timestamp: P: 1751032523619864 usec, L: 0
0627 13:55:47.891897 (+ 190us) write_op.cc:270] Start()
0627 13:55:47.891915 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523626732 usec, L: 0
0627 13:55:47.892077 (+ 162us) write_op.cc:270] Start()
0627 13:55:47.892095 (+ 18us) write_op.cc:276] Timestamp: P: 1751032523664671 usec, L: 0
0627 13:55:47.892246 (+ 151us) write_op.cc:270] Start()
0627 13:55:47.892263 (+ 17us) write_op.cc:276] Timestamp: P: 1751032523682725 usec, L: 0
0627 13:55:47.892403 (+ 140us) write_op.cc:270] Start()
0627 13:55:47.892419 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523685868 usec, L: 0
0627 13:55:47.892559 (+ 140us) write_op.cc:270] Start()
0627 13:55:47.892575 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523710220 usec, L: 0
0627 13:55:47.892710 (+ 135us) write_op.cc:270] Start()
0627 13:55:47.892732 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523717926 usec, L: 0
0627 13:55:47.892909 (+ 177us) write_op.cc:270] Start()
0627 13:55:47.892930 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523722325 usec, L: 0
0627 13:55:47.898532 (+ 5602us) write_op.cc:270] Start()
0627 13:55:47.898557 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523734584 usec, L: 0
0627 13:55:47.898790 (+ 233us) write_op.cc:270] Start()
0627 13:55:47.898958 (+ 168us) write_op.cc:276] Timestamp: P: 1751032523761189 usec, L: 0
0627 13:55:47.904546 (+ 5588us) write_op.cc:270] Start()
0627 13:55:47.904574 (+ 28us) write_op.cc:276] Timestamp: P: 1751032523769415 usec, L: 0
0627 13:55:47.904777 (+ 203us) write_op.cc:270] Start()
0627 13:55:47.904803 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523778430 usec, L: 0
0627 13:55:47.904995 (+ 192us) write_op.cc:270] Start()
0627 13:55:47.905041 (+ 46us) write_op.cc:276] Timestamp: P: 1751032523797323 usec, L: 0
0627 13:55:47.905229 (+ 188us) write_op.cc:270] Start()
0627 13:55:47.905250 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523813298 usec, L: 0
0627 13:55:47.905409 (+ 159us) write_op.cc:270] Start()
0627 13:55:47.905425 (+ 16us) write_op.cc:276] Timestamp: P: 1751032523821059 usec, L: 0
0627 13:55:47.907253 (+ 1828us) write_op.cc:270] Start()
0627 13:55:47.907279 (+ 26us) write_op.cc:276] Timestamp: P: 1751032523841250 usec, L: 0
0627 13:55:47.909351 (+ 2072us) write_op.cc:270] Start()
0627 13:55:47.909389 (+ 38us) write_op.cc:276] Timestamp: P: 1751032523856279 usec, L: 0
0627 13:55:47.923169 (+ 13780us) write_op.cc:270] Start()
0627 13:55:47.923198 (+ 29us) write_op.cc:276] Timestamp: P: 1751032523856979 usec, L: 0
0627 13:55:47.923426 (+ 228us) write_op.cc:270] Start()
0627 13:55:47.923447 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523865112 usec, L: 0
0627 13:55:47.923784 (+ 337us) write_op.cc:270] Start()
0627 13:55:47.923806 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523894309 usec, L: 0
0627 13:55:47.924489 (+ 683us) write_op.cc:270] Start()
0627 13:55:47.924510 (+ 21us) write_op.cc:276] Timestamp: P: 1751032523913268 usec, L: 0
0627 13:55:47.925228 (+ 718us) write_op.cc:270] Start()
0627 13:55:47.925250 (+ 22us) write_op.cc:276] Timestamp: P: 1751032523914078 usec, L: 0
0627 13:55:47.925405 (+ 155us) write_op.cc:270] Start()
0627 13:55:47.925424 (+ 19us) write_op.cc:276] Timestamp: P: 1751032523919049 usec, L: 0
0627 13:55:47.928004 (+ 2580us) write_op.cc:270] Start()
0627 13:55:47.928029 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523946200 usec, L: 0
0627 13:55:47.928261 (+ 232us) write_op.cc:270] Start()
0627 13:55:47.930693 (+ 2432us) write_op.cc:276] Timestamp: P: 1751032523967813 usec, L: 0
0627 13:55:47.930898 (+ 205us) write_op.cc:270] Start()
0627 13:55:47.930923 (+ 25us) write_op.cc:276] Timestamp: P: 1751032523991855 usec, L: 0
0627 13:55:47.931107 (+ 184us) write_op.cc:270] Start()
0627 13:55:47.931126 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524000873 usec, L: 0
0627 13:55:47.931317 (+ 191us) write_op.cc:270] Start()
0627 13:55:47.931337 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524009345 usec, L: 0
0627 13:55:47.931496 (+ 159us) write_op.cc:270] Start()
0627 13:55:47.931516 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524030815 usec, L: 0
0627 13:55:47.931762 (+ 246us) write_op.cc:270] Start()
0627 13:55:47.931820 (+ 58us) write_op.cc:276] Timestamp: P: 1751032524040338 usec, L: 0
0627 13:55:47.953631 (+ 21811us) write_op.cc:270] Start()
0627 13:55:47.953659 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524053716 usec, L: 0
0627 13:55:47.969944 (+ 16285us) write_op.cc:270] Start()
0627 13:55:47.969972 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524085931 usec, L: 0
0627 13:55:47.975722 (+ 5750us) write_op.cc:270] Start()
0627 13:55:47.975750 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524087917 usec, L: 0
0627 13:55:47.985827 (+ 10077us) write_op.cc:270] Start()
0627 13:55:47.985860 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524092252 usec, L: 0
0627 13:55:47.986258 (+ 398us) write_op.cc:270] Start()
0627 13:55:47.986287 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524117260 usec, L: 0
0627 13:55:47.996979 (+ 10692us) write_op.cc:270] Start()
0627 13:55:47.997010 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524139398 usec, L: 0
0627 13:55:47.997232 (+ 222us) write_op.cc:270] Start()
0627 13:55:47.997258 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524145710 usec, L: 0
0627 13:55:48.001375 (+ 4117us) write_op.cc:270] Start()
0627 13:55:48.001404 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524149724 usec, L: 0
0627 13:55:48.006140 (+ 4736us) write_op.cc:270] Start()
0627 13:55:48.006168 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524162158 usec, L: 0
0627 13:55:48.009380 (+ 3212us) write_op.cc:270] Start()
0627 13:55:48.009407 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524182258 usec, L: 0
0627 13:55:48.015162 (+ 5755us) write_op.cc:270] Start()
0627 13:55:48.015193 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524189106 usec, L: 0
0627 13:55:48.043110 (+ 27917us) write_op.cc:270] Start()
0627 13:55:48.043513 (+ 403us) write_op.cc:276] Timestamp: P: 1751032524204064 usec, L: 0
0627 13:55:48.043747 (+ 234us) write_op.cc:270] Start()
0627 13:55:48.043770 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524224631 usec, L: 0
0627 13:55:48.044308 (+ 538us) write_op.cc:270] Start()
0627 13:55:48.044333 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524228273 usec, L: 0
0627 13:55:48.044506 (+ 173us) write_op.cc:270] Start()
0627 13:55:48.044528 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524242673 usec, L: 0
0627 13:55:48.044706 (+ 178us) write_op.cc:270] Start()
0627 13:55:48.044729 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524254636 usec, L: 0
0627 13:55:48.044903 (+ 174us) write_op.cc:270] Start()
0627 13:55:48.044956 (+ 53us) write_op.cc:276] Timestamp: P: 1751032524266717 usec, L: 0
0627 13:55:48.045133 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.045155 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524279899 usec, L: 0
0627 13:55:48.045350 (+ 195us) write_op.cc:270] Start()
0627 13:55:48.045373 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524293210 usec, L: 0
0627 13:55:48.045536 (+ 163us) write_op.cc:270] Start()
0627 13:55:48.045558 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524314544 usec, L: 0
0627 13:55:48.053505 (+ 7947us) write_op.cc:270] Start()
0627 13:55:48.053537 (+ 32us) write_op.cc:276] Timestamp: P: 1751032524321693 usec, L: 0
0627 13:55:48.055959 (+ 2422us) write_op.cc:270] Start()
0627 13:55:48.055992 (+ 33us) write_op.cc:276] Timestamp: P: 1751032524346887 usec, L: 0
0627 13:55:48.056234 (+ 242us) write_op.cc:270] Start()
0627 13:55:48.056258 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524358787 usec, L: 0
0627 13:55:48.056510 (+ 252us) write_op.cc:270] Start()
0627 13:55:48.056541 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524367781 usec, L: 0
0627 13:55:48.056734 (+ 193us) write_op.cc:270] Start()
0627 13:55:48.056755 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524375783 usec, L: 0
0627 13:55:48.056964 (+ 209us) write_op.cc:270] Start()
0627 13:55:48.056986 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524394875 usec, L: 0
0627 13:55:48.058478 (+ 1492us) write_op.cc:270] Start()
0627 13:55:48.058501 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524408364 usec, L: 0
0627 13:55:48.058693 (+ 192us) write_op.cc:270] Start()
0627 13:55:48.058712 (+ 19us) write_op.cc:276] Timestamp: P: 1751032524428550 usec, L: 0
0627 13:55:48.058868 (+ 156us) write_op.cc:270] Start()
0627 13:55:48.058885 (+ 17us) write_op.cc:276] Timestamp: P: 1751032524434460 usec, L: 0
0627 13:55:48.067299 (+ 8414us) write_op.cc:270] Start()
0627 13:55:48.067336 (+ 37us) write_op.cc:276] Timestamp: P: 1751032524461643 usec, L: 0
0627 13:55:48.070628 (+ 3292us) write_op.cc:270] Start()
0627 13:55:48.070657 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524464452 usec, L: 0
0627 13:55:48.078828 (+ 8171us) write_op.cc:270] Start()
0627 13:55:48.078854 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524480441 usec, L: 0
0627 13:55:48.095293 (+ 16439us) write_op.cc:270] Start()
0627 13:55:48.095324 (+ 31us) write_op.cc:276] Timestamp: P: 1751032524482406 usec, L: 0
0627 13:55:48.095824 (+ 500us) write_op.cc:270] Start()
0627 13:55:48.095853 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524514466 usec, L: 0
0627 13:55:48.097244 (+ 1391us) write_op.cc:270] Start()
0627 13:55:48.097272 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524515169 usec, L: 0
0627 13:55:48.097491 (+ 219us) write_op.cc:270] Start()
0627 13:55:48.097517 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524536296 usec, L: 0
0627 13:55:48.098388 (+ 871us) write_op.cc:270] Start()
0627 13:55:48.098411 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524564134 usec, L: 0
0627 13:55:48.114390 (+ 15979us) write_op.cc:270] Start()
0627 13:55:48.114417 (+ 27us) write_op.cc:276] Timestamp: P: 1751032524579427 usec, L: 0
0627 13:55:48.114591 (+ 174us) write_op.cc:270] Start()
0627 13:55:48.114611 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524581041 usec, L: 0
0627 13:55:48.117866 (+ 3255us) write_op.cc:270] Start()
0627 13:55:48.117900 (+ 34us) write_op.cc:276] Timestamp: P: 1751032524613519 usec, L: 0
0627 13:55:48.120423 (+ 2523us) write_op.cc:270] Start()
0627 13:55:48.120453 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524622337 usec, L: 0
0627 13:55:48.120674 (+ 221us) write_op.cc:270] Start()
0627 13:55:48.120698 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524640822 usec, L: 0
0627 13:55:48.120883 (+ 185us) write_op.cc:270] Start()
0627 13:55:48.120905 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524654615 usec, L: 0
0627 13:55:48.121317 (+ 412us) write_op.cc:270] Start()
0627 13:55:48.121340 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524666836 usec, L: 0
0627 13:55:48.122779 (+ 1439us) write_op.cc:270] Start()
0627 13:55:48.122803 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524692613 usec, L: 0
0627 13:55:48.136292 (+ 13489us) write_op.cc:270] Start()
0627 13:55:48.136333 (+ 41us) write_op.cc:276] Timestamp: P: 1751032524714753 usec, L: 0
0627 13:55:48.136631 (+ 298us) write_op.cc:270] Start()
0627 13:55:48.136653 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524719108 usec, L: 0
0627 13:55:48.144324 (+ 7671us) write_op.cc:270] Start()
0627 13:55:48.144352 (+ 28us) write_op.cc:276] Timestamp: P: 1751032524720680 usec, L: 0
0627 13:55:48.146705 (+ 2353us) write_op.cc:270] Start()
0627 13:55:48.146731 (+ 26us) write_op.cc:276] Timestamp: P: 1751032524748791 usec, L: 0
0627 13:55:48.146982 (+ 251us) write_op.cc:270] Start()
0627 13:55:48.147005 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524768414 usec, L: 0
0627 13:55:48.147182 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.147204 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524770487 usec, L: 0
0627 13:55:48.147391 (+ 187us) write_op.cc:270] Start()
0627 13:55:48.147412 (+ 21us) write_op.cc:276] Timestamp: P: 1751032524779533 usec, L: 0
0627 13:55:48.147587 (+ 175us) write_op.cc:270] Start()
0627 13:55:48.147610 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524804245 usec, L: 0
0627 13:55:48.147835 (+ 225us) write_op.cc:270] Start()
0627 13:55:48.147879 (+ 44us) write_op.cc:276] Timestamp: P: 1751032524815886 usec, L: 0
0627 13:55:48.148085 (+ 206us) write_op.cc:270] Start()
0627 13:55:48.148115 (+ 30us) write_op.cc:276] Timestamp: P: 1751032524822346 usec, L: 0
0627 13:55:48.148313 (+ 198us) write_op.cc:270] Start()
0627 13:55:48.148337 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524823723 usec, L: 0
0627 13:55:48.148542 (+ 205us) write_op.cc:270] Start()
0627 13:55:48.148564 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524864237 usec, L: 0
0627 13:55:48.148749 (+ 185us) write_op.cc:270] Start()
0627 13:55:48.148773 (+ 24us) write_op.cc:276] Timestamp: P: 1751032524874554 usec, L: 0
0627 13:55:48.148998 (+ 225us) write_op.cc:270] Start()
0627 13:55:48.149020 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524879722 usec, L: 0
0627 13:55:48.149244 (+ 224us) write_op.cc:270] Start()
0627 13:55:48.149266 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524880309 usec, L: 0
0627 13:55:48.149470 (+ 204us) write_op.cc:270] Start()
0627 13:55:48.149514 (+ 44us) write_op.cc:276] Timestamp: P: 1751032524916421 usec, L: 0
0627 13:55:48.150284 (+ 770us) write_op.cc:270] Start()
0627 13:55:48.150306 (+ 22us) write_op.cc:276] Timestamp: P: 1751032524929074 usec, L: 0
0627 13:55:48.150483 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.150503 (+ 20us) write_op.cc:276] Timestamp: P: 1751032524931092 usec, L: 0
0627 13:55:48.150742 (+ 239us) write_op.cc:270] Start()
0627 13:55:48.150765 (+ 23us) write_op.cc:276] Timestamp: P: 1751032524945413 usec, L: 0
0627 13:55:48.150957 (+ 192us) write_op.cc:270] Start()
0627 13:55:48.158506 (+ 7549us) write_op.cc:276] Timestamp: P: 1751032524954780 usec, L: 0
0627 13:55:48.158722 (+ 216us) write_op.cc:270] Start()
0627 13:55:48.158747 (+ 25us) write_op.cc:276] Timestamp: P: 1751032524983698 usec, L: 0
0627 13:55:48.158947 (+ 200us) write_op.cc:270] Start()
0627 13:55:48.158976 (+ 29us) write_op.cc:276] Timestamp: P: 1751032524991793 usec, L: 0
0627 13:55:48.159171 (+ 195us) write_op.cc:270] Start()
0627 13:55:48.159191 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525009366 usec, L: 0
0627 13:55:48.159377 (+ 186us) write_op.cc:270] Start()
0627 13:55:48.159399 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525022433 usec, L: 0
0627 13:55:48.159568 (+ 169us) write_op.cc:270] Start()
0627 13:55:48.159589 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525038903 usec, L: 0
0627 13:55:48.159766 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.159786 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525043501 usec, L: 0
0627 13:55:48.160025 (+ 239us) write_op.cc:270] Start()
0627 13:55:48.160045 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525056929 usec, L: 0
0627 13:55:48.160255 (+ 210us) write_op.cc:270] Start()
0627 13:55:48.160277 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525094423 usec, L: 0
0627 13:55:48.160470 (+ 193us) write_op.cc:270] Start()
0627 13:55:48.160491 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525104524 usec, L: 0
0627 13:55:48.160686 (+ 195us) write_op.cc:270] Start()
0627 13:55:48.160708 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525105349 usec, L: 0
0627 13:55:48.160883 (+ 175us) write_op.cc:270] Start()
0627 13:55:48.160908 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525128132 usec, L: 0
0627 13:55:48.161114 (+ 206us) write_op.cc:270] Start()
0627 13:55:48.161155 (+ 41us) write_op.cc:276] Timestamp: P: 1751032525156035 usec, L: 0
0627 13:55:48.161348 (+ 193us) write_op.cc:270] Start()
0627 13:55:48.161373 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525158826 usec, L: 0
0627 13:55:48.161543 (+ 170us) write_op.cc:270] Start()
0627 13:55:48.163061 (+ 1518us) write_op.cc:276] Timestamp: P: 1751032525162421 usec, L: 0
0627 13:55:48.163283 (+ 222us) write_op.cc:270] Start()
0627 13:55:48.163307 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525177863 usec, L: 0
0627 13:55:48.163509 (+ 202us) write_op.cc:270] Start()
0627 13:55:48.163533 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525209191 usec, L: 0
0627 13:55:48.163917 (+ 384us) write_op.cc:270] Start()
0627 13:55:48.163940 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525215456 usec, L: 0
0627 13:55:48.164154 (+ 214us) write_op.cc:270] Start()
0627 13:55:48.164176 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525217508 usec, L: 0
0627 13:55:48.164952 (+ 776us) write_op.cc:270] Start()
0627 13:55:48.164988 (+ 36us) write_op.cc:276] Timestamp: P: 1751032525241969 usec, L: 0
0627 13:55:48.165179 (+ 191us) write_op.cc:270] Start()
0627 13:55:48.165200 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525253873 usec, L: 0
0627 13:55:48.165403 (+ 203us) write_op.cc:270] Start()
0627 13:55:48.165427 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525281729 usec, L: 0
0627 13:55:48.165622 (+ 195us) write_op.cc:270] Start()
0627 13:55:48.165644 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525284418 usec, L: 0
0627 13:55:48.165837 (+ 193us) write_op.cc:270] Start()
0627 13:55:48.165903 (+ 66us) write_op.cc:276] Timestamp: P: 1751032525287655 usec, L: 0
0627 13:55:48.166177 (+ 274us) write_op.cc:270] Start()
0627 13:55:48.167259 (+ 1082us) write_op.cc:276] Timestamp: P: 1751032525303577 usec, L: 0
0627 13:55:48.167524 (+ 265us) write_op.cc:270] Start()
0627 13:55:48.167550 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525349650 usec, L: 0
0627 13:55:48.167832 (+ 282us) write_op.cc:270] Start()
0627 13:55:48.167855 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525352186 usec, L: 0
0627 13:55:48.168067 (+ 212us) write_op.cc:270] Start()
0627 13:55:48.168094 (+ 27us) write_op.cc:276] Timestamp: P: 1751032525353218 usec, L: 0
0627 13:55:48.168261 (+ 167us) write_op.cc:270] Start()
0627 13:55:48.168299 (+ 38us) write_op.cc:276] Timestamp: P: 1751032525354402 usec, L: 0
0627 13:55:48.168483 (+ 184us) write_op.cc:270] Start()
0627 13:55:48.168504 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525376546 usec, L: 0
0627 13:55:48.168693 (+ 189us) write_op.cc:270] Start()
0627 13:55:48.168713 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525390551 usec, L: 0
0627 13:55:48.168937 (+ 224us) write_op.cc:270] Start()
0627 13:55:48.168959 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525398853 usec, L: 0
0627 13:55:48.170476 (+ 1517us) write_op.cc:270] Start()
0627 13:55:48.170499 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525437043 usec, L: 0
0627 13:55:48.185940 (+ 15441us) write_op.cc:270] Start()
0627 13:55:48.185966 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525447157 usec, L: 0
0627 13:55:48.190811 (+ 4845us) write_op.cc:270] Start()
0627 13:55:48.190840 (+ 29us) write_op.cc:276] Timestamp: P: 1751032525448265 usec, L: 0
0627 13:55:48.191065 (+ 225us) write_op.cc:270] Start()
0627 13:55:48.191087 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525448899 usec, L: 0
0627 13:55:48.191286 (+ 199us) write_op.cc:270] Start()
0627 13:55:48.191308 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525480829 usec, L: 0
0627 13:55:48.191502 (+ 194us) write_op.cc:270] Start()
0627 13:55:48.191523 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525487028 usec, L: 0
0627 13:55:48.191704 (+ 181us) write_op.cc:270] Start()
0627 13:55:48.191724 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525497869 usec, L: 0
0627 13:55:48.191901 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.191922 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525503027 usec, L: 0
0627 13:55:48.192128 (+ 206us) write_op.cc:270] Start()
0627 13:55:48.192150 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525514070 usec, L: 0
0627 13:55:48.192958 (+ 808us) write_op.cc:270] Start()
0627 13:55:48.192981 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525566992 usec, L: 0
0627 13:55:48.194873 (+ 1892us) write_op.cc:270] Start()
0627 13:55:48.194899 (+ 26us) write_op.cc:276] Timestamp: P: 1751032525570221 usec, L: 0
0627 13:55:48.196362 (+ 1463us) write_op.cc:270] Start()
0627 13:55:48.196385 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525586105 usec, L: 0
0627 13:55:48.197033 (+ 648us) write_op.cc:270] Start()
0627 13:55:48.197072 (+ 39us) write_op.cc:276] Timestamp: P: 1751032525593533 usec, L: 0
0627 13:55:48.202925 (+ 5853us) write_op.cc:270] Start()
0627 13:55:48.202950 (+ 25us) write_op.cc:276] Timestamp: P: 1751032525620744 usec, L: 0
0627 13:55:48.203123 (+ 173us) write_op.cc:270] Start()
0627 13:55:48.203147 (+ 24us) write_op.cc:276] Timestamp: P: 1751032525634400 usec, L: 0
0627 13:55:48.203324 (+ 177us) write_op.cc:270] Start()
0627 13:55:48.203345 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525636802 usec, L: 0
0627 13:55:48.203512 (+ 167us) write_op.cc:270] Start()
0627 13:55:48.203532 (+ 20us) write_op.cc:276] Timestamp: P: 1751032525648552 usec, L: 0
0627 13:55:48.203718 (+ 186us) write_op.cc:270] Start()
0627 13:55:48.203739 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525685171 usec, L: 0
0627 13:55:48.203899 (+ 160us) write_op.cc:270] Start()
0627 13:55:48.203922 (+ 23us) write_op.cc:276] Timestamp: P: 1751032525707239 usec, L: 0
0627 13:55:48.204126 (+ 204us) write_op.cc:270] Start()
0627 13:55:48.204147 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525715984 usec, L: 0
0627 13:55:48.204302 (+ 155us) write_op.cc:270] Start()
0627 13:55:48.204323 (+ 21us) write_op.cc:276] Timestamp: P: 1751032525722263 usec, L: 0
0627 13:55:48.204502 (+ 179us) write_op.cc:270] Start()
0627 13:55:48.204524 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525740833 usec, L: 0
0627 13:55:48.204679 (+ 155us) write_op.cc:270] Start()
0627 13:55:48.204701 (+ 22us) write_op.cc:276] Timestamp: P: 1751032525764163 usec, L: 0
0627 13:55:48.210575 (+ 5874us) write_op.cc:270] Start()
0627 13:55:48.210600 (+ 25us) write_op.cc:276] Timestamp: P: 1751
I20250627 13:55:51.853832 9920 raft_consensus.cc:491] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:55:51.854003 9920 raft_consensus.cc:513] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } }
I20250627 13:55:51.854816 9920 leader_election.cc:290] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:51.855194 9174 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1598 } } ignore_live_leader: false dest_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" is_pre_election: true
I20250627 13:55:51.856102 9300 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab269b1c09ed41b0b51072f18c2462d8" candidate_uuid: "18c6bddc499b4363a95de5d2578e4a07" candidate_term: 2 candidate_status { last_received { term: 1 index: 1598 } } ignore_live_leader: false dest_uuid: "8064817311e242618056800a0ca537dd" is_pre_election: true
I20250627 13:55:51.857223 9760 leader_election.cc:304] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18c6bddc499b4363a95de5d2578e4a07; no voters: 8064817311e242618056800a0ca537dd, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:55:51.859225 9920 raft_consensus.cc:2747] T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250627 13:55:52.780408 9948 log.cc:927] Time spent T ab269b1c09ed41b0b51072f18c2462d8 P 18c6bddc499b4363a95de5d2578e4a07: Append to log took a long time: real 0.062s user 0.004s sys 0.000s
I20250627 13:55:53.166821 9675 ts_manager.cc:284] Unset tserver state for 18c6bddc499b4363a95de5d2578e4a07 from MAINTENANCE_MODE
I20250627 13:55:53.174651 9219 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:53.342355 9349 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:53.810248 9599 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:53.864068 9883 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:56.466549 9675 ts_manager.cc:295] Set tserver state for 18c6bddc499b4363a95de5d2578e4a07 to MAINTENANCE_MODE
I20250627 13:55:56.470495 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 9743
I20250627 13:55:56.471704 9878 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:55:57.166386 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 9743
W20250627 13:55:57.298177 9239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.167.193:45719 (error 108) [suppressed 36 similar messages]
W20250627 13:55:57.305497 9109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.167.193:45719 (error 108) [suppressed 32 similar messages]
W20250627 13:55:57.314216 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.314514 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.314704 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.321758 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.322010 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.322362 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:57.728444 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:57.763960 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:57.771198 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:57.777839 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:57.789454 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:57.816174 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:58.242470 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:58.268496 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:58.273379 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:58.320464 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:58.326809 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:58.391039 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250627 13:55:58.506464 9925 consensus_queue.cc:579] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.047s)
I20250627 13:55:58.523505 9975 consensus_queue.cc:579] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.043s)
I20250627 13:55:58.524551 9996 consensus_queue.cc:579] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.025s)
I20250627 13:55:58.546814 9925 consensus_queue.cc:579] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.047s)
W20250627 13:55:58.727147 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:58.767597 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:58.788652 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:58.844448 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:58.865231 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:55:58.921568 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250627 13:55:59.191147 10002 consensus_queue.cc:579] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.033s)
I20250627 13:55:59.199081 10001 consensus_queue.cc:579] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.034s)
W20250627 13:55:59.268464 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:59.296739 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:59.332083 9674 ts_manager.cc:284] Unset tserver state for 18c6bddc499b4363a95de5d2578e4a07 from MAINTENANCE_MODE
W20250627 13:55:59.334985 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:59.393669 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:59.414543 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:55:59.419950 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:59.562765 9349 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:59.606212 9303 consensus_queue.cc:237] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2047, Committed index: 2047, Last appended: 1.2047, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
W20250627 13:55:59.611351 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.611588 9303 consensus_queue.cc:237] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2048, Committed index: 2048, Last appended: 2.2049, Last appended by leader: 160, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2050 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:55:59.614006 9300 consensus_queue.cc:237] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2047, Committed index: 2047, Last appended: 1.2047, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:55:59.626803 9172 raft_consensus.cc:1273] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Refusing update from remote peer 8064817311e242618056800a0ca537dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2047. Preceding OpId from leader: term: 1 index: 2048. (index mismatch)
I20250627 13:55:59.627233 9174 raft_consensus.cc:1273] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 2 FOLLOWER]: Refusing update from remote peer 8064817311e242618056800a0ca537dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2049. Preceding OpId from leader: term: 2 index: 2050. (index mismatch)
I20250627 13:55:59.632655 9925 consensus_queue.cc:1035] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2048, Last known committed idx: 2047, Time since last communication: 0.000s
W20250627 13:55:59.636624 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.637604 9925 consensus_queue.cc:1035] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2050, Last known committed idx: 2048, Time since last communication: 0.000s
I20250627 13:55:59.637912 9174 raft_consensus.cc:1273] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Refusing update from remote peer 8064817311e242618056800a0ca537dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2047. Preceding OpId from leader: term: 1 index: 2050. (index mismatch)
I20250627 13:55:59.640798 9925 consensus_queue.cc:1035] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2048, Last known committed idx: 2047, Time since last communication: 0.000s
W20250627 13:55:59.649338 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.651862 9996 raft_consensus.cc:2953] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [term 1 LEADER]: Committing config change with OpId 1.2048: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.660135 9661 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ab269b1c09ed41b0b51072f18c2462d8 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250627 13:55:59.660975 9237 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab269b1c09ed41b0b51072f18c2462d8. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:59.661168 9237 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ef29c61340f4869b8703e67f2e32f14. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:59.661324 9237 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f55dc8052c494b8d8a79c2e7f9ed39d4. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.666406 9925 raft_consensus.cc:2953] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [term 2 LEADER]: Committing config change with OpId 2.2050: config changed from index -1 to 2050, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2050 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.666435 9674 catalog_manager.cc:5582] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd reported cstate change: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 1 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.667258 9174 raft_consensus.cc:2953] T ab269b1c09ed41b0b51072f18c2462d8 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Committing config change with OpId 1.2048: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.670462 9999 raft_consensus.cc:2953] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [term 1 LEADER]: Committing config change with OpId 1.2048: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.674453 9661 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:55:59.681605 9661 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ef29c61340f4869b8703e67f2e32f14 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:55:59.681512 9174 raft_consensus.cc:2953] T 9ef29c61340f4869b8703e67f2e32f14 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 2 FOLLOWER]: Committing config change with OpId 2.2050: config changed from index -1 to 2050, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2050 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.684203 9674 catalog_manager.cc:5582] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd reported cstate change: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 1 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.684909 9674 catalog_manager.cc:5582] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd reported cstate change: config changed from index -1 to 2050, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 2 leader_uuid: "8064817311e242618056800a0ca537dd" committed_config { opid_index: 2050 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.687359 9172 raft_consensus.cc:2953] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Committing config change with OpId 1.2048: config changed from index -1 to 2048, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.697043 9219 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:59.728020 10015 ts_tablet_manager.cc:927] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:59.736486 10014 ts_tablet_manager.cc:927] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:59.737684 10017 ts_tablet_manager.cc:927] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:55:59.755266 10015 tablet_copy_client.cc:323] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.195:43925
I20250627 13:55:59.771660 10017 tablet_copy_client.cc:323] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.195:43925
I20250627 13:55:59.779909 9323 tablet_copy_service.cc:140] P 8064817311e242618056800a0ca537dd: Received BeginTabletCopySession request for tablet f55dc8052c494b8d8a79c2e7f9ed39d4 from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:54423)
I20250627 13:55:59.780215 9323 tablet_copy_service.cc:161] P 8064817311e242618056800a0ca537dd: Beginning new tablet copy session on tablet f55dc8052c494b8d8a79c2e7f9ed39d4 from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:54423: session id = 5fe0be3d9258445cbdc65ddc3a59f144-f55dc8052c494b8d8a79c2e7f9ed39d4
I20250627 13:55:59.780375 10014 tablet_copy_client.cc:323] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.195:43925
I20250627 13:55:59.780851 9324 tablet_copy_service.cc:140] P 8064817311e242618056800a0ca537dd: Received BeginTabletCopySession request for tablet 9ef29c61340f4869b8703e67f2e32f14 from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:54423)
I20250627 13:55:59.781112 9324 tablet_copy_service.cc:161] P 8064817311e242618056800a0ca537dd: Beginning new tablet copy session on tablet 9ef29c61340f4869b8703e67f2e32f14 from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:54423: session id = 5fe0be3d9258445cbdc65ddc3a59f144-9ef29c61340f4869b8703e67f2e32f14
I20250627 13:55:59.782315 9322 tablet_copy_service.cc:140] P 8064817311e242618056800a0ca537dd: Received BeginTabletCopySession request for tablet ab269b1c09ed41b0b51072f18c2462d8 from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:54423)
I20250627 13:55:59.782519 9322 tablet_copy_service.cc:161] P 8064817311e242618056800a0ca537dd: Beginning new tablet copy session on tablet ab269b1c09ed41b0b51072f18c2462d8 from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:54423: session id = 5fe0be3d9258445cbdc65ddc3a59f144-ab269b1c09ed41b0b51072f18c2462d8
I20250627 13:55:59.784456 9171 consensus_queue.cc:237] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2055, Committed index: 2055, Last appended: 1.2055, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:55:59.784634 9323 tablet_copy_source_session.cc:215] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:55:59.785609 9322 tablet_copy_source_session.cc:215] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:55:59.787739 9324 tablet_copy_source_session.cc:215] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:55:59.789088 9174 consensus_queue.cc:237] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2055, Committed index: 2055, Last appended: 1.2056, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:55:59.789470 9171 consensus_queue.cc:237] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2054, Committed index: 2054, Last appended: 1.2055, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:55:59.793524 10015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ef29c61340f4869b8703e67f2e32f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:55:59.794694 10017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55dc8052c494b8d8a79c2e7f9ed39d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:55:59.798687 9107 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22aa1c36043f4e8781bac641bd1035aa. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.798900 10014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab269b1c09ed41b0b51072f18c2462d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250627 13:55:59.798935 9107 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c370dad30118471cbdc7094f7209aad5. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:59.805411 9107 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82aebc7eff8c4c8a86eb994bf24bd99c. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.806236 9301 raft_consensus.cc:1273] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Refusing update from remote peer e07fa0ee9b1b4e28b800d1ac6da68e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 2055. Preceding OpId from leader: term: 1 index: 2056. (index mismatch)
I20250627 13:55:59.807799 9304 raft_consensus.cc:1273] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Refusing update from remote peer e07fa0ee9b1b4e28b800d1ac6da68e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 2055. Preceding OpId from leader: term: 1 index: 2056. (index mismatch)
I20250627 13:55:59.810626 9968 consensus_queue.cc:1035] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2056, Last known committed idx: 2055, Time since last communication: 0.000s
I20250627 13:55:59.811201 9968 consensus_queue.cc:1035] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2056, Last known committed idx: 2054, Time since last communication: 0.000s
I20250627 13:55:59.815308 9302 raft_consensus.cc:1273] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Refusing update from remote peer e07fa0ee9b1b4e28b800d1ac6da68e17: Log matching property violated. Preceding OpId in replica: term: 1 index: 2056. Preceding OpId from leader: term: 1 index: 2057. (index mismatch)
W20250627 13:55:59.817272 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:59.817410 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:59.817497 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:55:59.824556 10001 consensus_queue.cc:1035] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2057, Last known committed idx: 2055, Time since last communication: 0.000s
I20250627 13:55:59.825151 10001 raft_consensus.cc:2953] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Committing config change with OpId 1.2056: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.825292 9951 raft_consensus.cc:2953] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Committing config change with OpId 1.2056: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.830976 9662 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 82aebc7eff8c4c8a86eb994bf24bd99c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:55:59.836058 9304 raft_consensus.cc:2953] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Committing config change with OpId 1.2056: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.839054 10014 tablet_copy_client.cc:806] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:55:59.838979 9674 catalog_manager.cc:5582] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.839392 10014 tablet_copy_client.cc:670] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:55:59.839721 9662 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c370dad30118471cbdc7094f7209aad5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:55:59.842197 10015 tablet_copy_client.cc:806] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:55:59.842456 10015 tablet_copy_client.cc:670] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:55:59.844477 9302 raft_consensus.cc:2953] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Committing config change with OpId 1.2056: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.850394 9599 heartbeater.cc:507] Master 127.8.167.254:41403 requested a full tablet report, sending...
I20250627 13:55:59.850417 10017 tablet_copy_client.cc:806] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:55:59.851438 10017 tablet_copy_client.cc:670] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:55:59.853114 9951 raft_consensus.cc:2953] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 LEADER]: Committing config change with OpId 1.2057: config changed from index -1 to 2057, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.855227 9674 catalog_manager.cc:5582] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: config changed from index -1 to 2056, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.858012 9300 raft_consensus.cc:2953] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Committing config change with OpId 1.2057: config changed from index -1 to 2057, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New config: { opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } } }
I20250627 13:55:59.864957 9674 catalog_manager.cc:5582] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 reported cstate change: config changed from index -1 to 2057, NON_VOTER 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196) added. New cstate: current_term: 1 leader_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" committed_config { opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250627 13:55:59.870623 9662 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 22aa1c36043f4e8781bac641bd1035aa with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250627 13:55:59.880990 10014 tablet_copy_client.cc:538] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:55:59.885855 10014 tablet_bootstrap.cc:492] T ab269b1c09ed41b0b51072f18c2462d8 P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
I20250627 13:55:59.895676 10015 tablet_copy_client.cc:538] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:55:59.943348 10017 tablet_copy_client.cc:538] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:55:59.951113 10015 tablet_bootstrap.cc:492] T 9ef29c61340f4869b8703e67f2e32f14 P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
I20250627 13:55:59.951687 10025 ts_tablet_manager.cc:927] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:59.960806 10017 tablet_bootstrap.cc:492] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
I20250627 13:55:59.969300 10025 tablet_copy_client.cc:323] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.194:35625
I20250627 13:55:59.979408 10023 ts_tablet_manager.cc:927] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:59.985751 10024 ts_tablet_manager.cc:927] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: Initiating tablet copy from peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:55:59.991334 10023 tablet_copy_client.cc:323] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.194:35625
I20250627 13:56:00.001965 10024 tablet_copy_client.cc:323] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Beginning tablet copy session from remote peer at address 127.8.167.194:35625
I20250627 13:56:00.013293 9194 tablet_copy_service.cc:140] P e07fa0ee9b1b4e28b800d1ac6da68e17: Received BeginTabletCopySession request for tablet c370dad30118471cbdc7094f7209aad5 from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:55787)
I20250627 13:56:00.013509 9194 tablet_copy_service.cc:161] P e07fa0ee9b1b4e28b800d1ac6da68e17: Beginning new tablet copy session on tablet c370dad30118471cbdc7094f7209aad5 from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:55787: session id = 5fe0be3d9258445cbdc65ddc3a59f144-c370dad30118471cbdc7094f7209aad5
I20250627 13:56:00.014194 9192 tablet_copy_service.cc:140] P e07fa0ee9b1b4e28b800d1ac6da68e17: Received BeginTabletCopySession request for tablet 22aa1c36043f4e8781bac641bd1035aa from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:55787)
I20250627 13:56:00.014328 9192 tablet_copy_service.cc:161] P e07fa0ee9b1b4e28b800d1ac6da68e17: Beginning new tablet copy session on tablet 22aa1c36043f4e8781bac641bd1035aa from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:55787: session id = 5fe0be3d9258445cbdc65ddc3a59f144-22aa1c36043f4e8781bac641bd1035aa
I20250627 13:56:00.017715 9192 tablet_copy_source_session.cc:215] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:56:00.013293 9193 tablet_copy_service.cc:140] P e07fa0ee9b1b4e28b800d1ac6da68e17: Received BeginTabletCopySession request for tablet 82aebc7eff8c4c8a86eb994bf24bd99c from peer 5fe0be3d9258445cbdc65ddc3a59f144 ({username='slave'} at 127.8.167.196:55787)
I20250627 13:56:00.019762 9193 tablet_copy_service.cc:161] P e07fa0ee9b1b4e28b800d1ac6da68e17: Beginning new tablet copy session on tablet 82aebc7eff8c4c8a86eb994bf24bd99c from peer 5fe0be3d9258445cbdc65ddc3a59f144 at {username='slave'} at 127.8.167.196:55787: session id = 5fe0be3d9258445cbdc65ddc3a59f144-82aebc7eff8c4c8a86eb994bf24bd99c
I20250627 13:56:00.026468 9194 tablet_copy_source_session.cc:215] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:56:00.029567 10025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c370dad30118471cbdc7094f7209aad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:00.037140 9193 tablet_copy_source_session.cc:215] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 13:56:00.038664 10023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82aebc7eff8c4c8a86eb994bf24bd99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:00.045194 10024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aa1c36043f4e8781bac641bd1035aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:00.062638 10025 tablet_copy_client.cc:806] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:56:00.063060 10025 tablet_copy_client.cc:670] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:56:00.063994 10024 tablet_copy_client.cc:806] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:56:00.064369 10024 tablet_copy_client.cc:670] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:56:00.068084 10023 tablet_copy_client.cc:806] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 0 data blocks...
I20250627 13:56:00.068498 10023 tablet_copy_client.cc:670] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Starting download of 1 WAL segments...
I20250627 13:56:00.075145 10025 tablet_copy_client.cc:538] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:56:00.081204 10023 tablet_copy_client.cc:538] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:56:00.089483 10024 tablet_copy_client.cc:538] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 13:56:00.090332 10025 tablet_bootstrap.cc:492] T c370dad30118471cbdc7094f7209aad5 P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
I20250627 13:56:00.093443 10023 tablet_bootstrap.cc:492] T 82aebc7eff8c4c8a86eb994bf24bd99c P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
I20250627 13:56:00.093585 10024 tablet_bootstrap.cc:492] T 22aa1c36043f4e8781bac641bd1035aa P 5fe0be3d9258445cbdc65ddc3a59f144: Bootstrap starting.
W20250627 13:56:00.126261 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.141010 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.185372 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.274145 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.283819 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.319841 9237 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.320425 9237 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.328574 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.342594 9237 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.460778 9107 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.473536 9107 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.551851 9107 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:00.654157 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.658355 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.678547 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.733286 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.765483 9237 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.774180 9109 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.778993 9237 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.807895 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.862670 9237 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:00.957419 9107 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:01.019125 9107 consensus_peers.cc:487] T c370dad30118471cbdc7094f7209aad5 P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:01.038689 9107 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:01.131003 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:01.178778 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:01.202672 9109 consensus_peers.cc:487] T 82aebc7eff8c4c8a86eb994bf24bd99c P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250627 13:56:01.238271 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 9092
I20250627 13:56:01.239467 9214 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:56:01.241986 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:01.256943 9237 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:01.260505 9237 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250627 13:56:01.319285 10017 log.cc:826] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 5fe0be3d9258445cbdc65ddc3a59f144: Log is configured to *not* fsync() on all Append() calls
W20250627 13:56:01.430487 9237 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250627 13:56:01.475598 9997 raft_consensus.cc:491] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:56:01.475818 9997 raft_consensus.cc:513] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:56:01.476627 9997 leader_election.cc:290] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
W20250627 13:56:01.478418 9239 leader_election.cc:336] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
I20250627 13:56:01.551254 9997 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:56:01.551431 9997 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:56:01.552011 9997 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
W20250627 13:56:01.553902 9239 leader_election.cc:336] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
I20250627 13:56:01.554179 9997 raft_consensus.cc:491] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:56:01.554311 9997 raft_consensus.cc:513] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2056 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:56:01.554821 9997 leader_election.cc:290] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
W20250627 13:56:01.556743 9239 leader_election.cc:336] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111)
I20250627 13:56:01.634523 9997 consensus_queue.cc:579] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.028s)
I20250627 13:56:01.667872 10046 consensus_queue.cc:579] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.052s)
W20250627 13:56:01.687296 9239 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:56:01.717214 10046 consensus_queue.cc:579] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.105s)
W20250627 13:56:01.719619 9239 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:56:01.758114 9237 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250627 13:56:01.760345 9239 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250627 13:56:01.770709 9237 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250627 13:56:01.861168 9109 consensus_peers.cc:487] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 -> Peer 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719): Couldn't send request to peer 18c6bddc499b4363a95de5d2578e4a07. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:45719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:01.861305 9109 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.193:45719, user_credentials={real_user=slave}} blocked reactor thread for 599528us
I20250627 13:56:01.861485 9951 raft_consensus.cc:491] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8064817311e242618056800a0ca537dd)
I20250627 13:56:01.861655 9951 raft_consensus.cc:513] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2048 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:56:01.862401 9951 leader_election.cc:290] T f55dc8052c494b8d8a79c2e7f9ed39d4 P e07fa0ee9b1b4e28b800d1ac6da68e17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), 8064817311e242618056800a0ca537dd (127.8.167.195:43925)
I20250627 13:56:01.863291 9951 consensus_queue.cc:579] T 22aa1c36043f4e8781bac641bd1035aa P e07fa0ee9b1b4e28b800d1ac6da68e17 [LEADER]: Leader has been unable to successfully communicate with peer 18c6bddc499b4363a95de5d2578e4a07 for more than 2 seconds (2.074s)
I20250627 13:56:01.863714 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 9092
W20250627 13:56:01.918414 9237 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer 5fe0be3d9258445cbdc65ddc3a59f144 (127.8.167.196:34697): Couldn't send request to peer 5fe0be3d9258445cbdc65ddc3a59f144. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250627 13:56:01.944484 9240 connection.cc:537] client connection to 127.8.167.194:35625 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:56:01.944653 9240 leader_election.cc:336] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:56:01.944762 9240 leader_election.cc:304] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8064817311e242618056800a0ca537dd; no voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17
W20250627 13:56:01.945214 9240 leader_election.cc:336] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:56:01.945334 9240 leader_election.cc:304] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8064817311e242618056800a0ca537dd; no voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17
I20250627 13:56:01.945617 10048 raft_consensus.cc:2747] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:56:01.945840 10048 raft_consensus.cc:2747] T 82aebc7eff8c4c8a86eb994bf24bd99c P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:56:01.947676 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 9222
W20250627 13:56:01.948503 9993 negotiation.cc:337] Failed RPC negotiation. Trace:
0627 13:56:01.946522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.167.194:35625 (local address 127.8.167.195:47667)
0627 13:56:01.946685 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0627 13:56:01.946706 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0627 13:56:01.946838 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0627 13:56:01.947596 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.167.194:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":39}
I20250627 13:56:01.948783 9344 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:56:02.457233 9240 leader_election.cc:336] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Network error: Client connection negotiation failed: client connection to 127.8.167.194:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:56:02.457430 9240 leader_election.cc:304] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8064817311e242618056800a0ca537dd; no voters: 18c6bddc499b4363a95de5d2578e4a07, e07fa0ee9b1b4e28b800d1ac6da68e17
W20250627 13:56:02.457791 9240 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.167.194:35625, user_credentials={real_user=slave}} blocked reactor thread for 508478us
W20250627 13:56:02.457945 9240 consensus_peers.cc:487] T ab269b1c09ed41b0b51072f18c2462d8 P 8064817311e242618056800a0ca537dd -> Peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Couldn't send request to peer e07fa0ee9b1b4e28b800d1ac6da68e17. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.194:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:56:02.458068 9240 consensus_peers.cc:487] T f55dc8052c494b8d8a79c2e7f9ed39d4 P 8064817311e242618056800a0ca537dd -> Peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Couldn't send request to peer e07fa0ee9b1b4e28b800d1ac6da68e17. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.194:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:56:02.458163 9240 consensus_peers.cc:487] T 9ef29c61340f4869b8703e67f2e32f14 P 8064817311e242618056800a0ca537dd -> Peer e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625): Couldn't send request to peer e07fa0ee9b1b4e28b800d1ac6da68e17. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.194:35625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:56:02.458729 10048 raft_consensus.cc:2747] T c370dad30118471cbdc7094f7209aad5 P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250627 13:56:02.459789 9997 raft_consensus.cc:491] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e07fa0ee9b1b4e28b800d1ac6da68e17)
I20250627 13:56:02.459946 9997 raft_consensus.cc:513] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2057 OBSOLETE_local: false peers { permanent_uuid: "18c6bddc499b4363a95de5d2578e4a07" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 45719 } } peers { permanent_uuid: "e07fa0ee9b1b4e28b800d1ac6da68e17" member_type: VOTER last_known_addr { host: "127.8.167.194" port: 35625 } } peers { permanent_uuid: "8064817311e242618056800a0ca537dd" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 43925 } } peers { permanent_uuid: "5fe0be3d9258445cbdc65ddc3a59f144" member_type: NON_VOTER last_known_addr { host: "127.8.167.196" port: 34697 } attrs { promote: true } }
I20250627 13:56:02.481241 9997 leader_election.cc:290] T 22aa1c36043f4e8781bac641bd1035aa P 8064817311e242618056800a0ca537dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6bddc499b4363a95de5d2578e4a07 (127.8.167.193:45719), e07fa0ee9b1b4e28b800d1ac6da68e17 (127.8.167.194:35625)
I20250627 13:56:02.514534 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 9222
I20250627 13:56:02.608757 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 9426
I20250627 13:56:02.610116 9594 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:02.826180 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 9426
I20250627 13:56:02.857662 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 9642
I20250627 13:56:02.858925 9706 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:02.965624 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 9642
2025-06-27T13:56:02Z chronyd exiting
[ OK ] MaintenanceModeRF3ITest.TestFailedTServerInMaintenanceModeDoesntRereplicate (46966 ms)
[----------] 1 test from MaintenanceModeRF3ITest (46966 ms total)
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4
2025-06-27T13:56:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:56:03Z Disabled control of system clock
I20250627 13:56:03.070755 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:38489
--webserver_interface=127.8.167.254
--webserver_port=0
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.167.254:38489
--location_mapping_cmd=/tmp/dist-test-taskHxmGc1/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/location-assignment.state --map /L0:4
--master_client_location_assignment_enabled=false with env {}
W20250627 13:56:03.408190 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:03.408545 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:03.408684 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:03.416814 10064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:56:03.416937 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:03.417007 10064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:56:03.417088 10064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:56:03.426828 10064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskHxmGc1/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/location-assignment.state --map /L0:4
--ipki_ca_key_size=768
--master_addresses=127.8.167.254:38489
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.167.254:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.167.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:03.428587 10064 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:03.430502 10064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250627 13:56:03.441675 10064 server_base.cc:1048] running on GCE node
W20250627 13:56:03.441634 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:03.441578 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:03.441798 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:03.442878 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:03.444340 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:03.445556 10064 hybrid_clock.cc:648] HybridClock initialized: now 1751032563445504 us; error 37 us; skew 500 ppm
I20250627 13:56:03.445890 10064 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:03.448370 10064 webserver.cc:469] Webserver started at http://127.8.167.254:33327/ using document root <none> and password file <none>
I20250627 13:56:03.449158 10064 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:03.449306 10064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:03.449687 10064 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:56:03.452172 10064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data/instance:
uuid: "85e4da409af145278ba10cc404ad4dbb"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:03.452971 10064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal/instance:
uuid: "85e4da409af145278ba10cc404ad4dbb"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:03.458406 10064 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20250627 13:56:03.461649 10078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:03.462970 10064 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:56:03.463133 10064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal
uuid: "85e4da409af145278ba10cc404ad4dbb"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:03.463335 10064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:03.478823 10064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:03.479727 10064 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:03.480018 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:03.500507 10064 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.254:38489
I20250627 13:56:03.500536 10129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.254:38489 every 8 connection(s)
I20250627 13:56:03.502018 10064 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/data/info.pb
I20250627 13:56:03.506137 10130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:03.509086 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10064
I20250627 13:56:03.509383 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/master-0/wal/instance
I20250627 13:56:03.518188 10130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb: Bootstrap starting.
I20250627 13:56:03.521639 10130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb: Neither blocks nor log segments found. Creating new log.
I20250627 13:56:03.523249 10130 log.cc:826] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:03.526288 10130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb: No bootstrap required, opened a new log
I20250627 13:56:03.531637 10130 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } }
I20250627 13:56:03.531960 10130 raft_consensus.cc:383] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:56:03.532090 10130 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e4da409af145278ba10cc404ad4dbb, State: Initialized, Role: FOLLOWER
I20250627 13:56:03.532704 10130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } }
I20250627 13:56:03.532917 10130 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:56:03.533035 10130 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:56:03.533202 10130 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:56:03.535578 10130 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } }
I20250627 13:56:03.536109 10130 leader_election.cc:304] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e4da409af145278ba10cc404ad4dbb; no voters:
I20250627 13:56:03.536690 10130 leader_election.cc:290] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:56:03.536829 10135 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:56:03.537544 10135 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [term 1 LEADER]: Becoming Leader. State: Replica: 85e4da409af145278ba10cc404ad4dbb, State: Running, Role: LEADER
I20250627 13:56:03.538100 10135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } }
I20250627 13:56:03.538888 10130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:56:03.541165 10137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e4da409af145278ba10cc404ad4dbb. Latest consensus state: current_term: 1 leader_uuid: "85e4da409af145278ba10cc404ad4dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } } }
I20250627 13:56:03.541283 10136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e4da409af145278ba10cc404ad4dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e4da409af145278ba10cc404ad4dbb" member_type: VOTER last_known_addr { host: "127.8.167.254" port: 38489 } } }
I20250627 13:56:03.541569 10136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [sys.catalog]: This master's current role is: LEADER
I20250627 13:56:03.541576 10137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb [sys.catalog]: This master's current role is: LEADER
I20250627 13:56:03.543584 10143 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:56:03.548087 10143 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:56:03.557221 10143 catalog_manager.cc:1349] Generated new cluster ID: 6b6ee4d243954209913e77cb678554ed
I20250627 13:56:03.557415 10143 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:56:03.588456 10143 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:56:03.590103 10143 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:56:03.598906 10143 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 85e4da409af145278ba10cc404ad4dbb: Generated new TSK 0
I20250627 13:56:03.599825 10143 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:56:03.606202 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:0
--local_ip_for_outbound_sockets=127.8.167.193
--webserver_interface=127.8.167.193
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:03.945753 10154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:03.946153 10154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:03.946317 10154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:03.954334 10154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:03.954576 10154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:56:03.964279 10154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:03.965994 10154 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:03.967942 10154 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:03.980089 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:03.980191 10160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:03.980134 10159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:03.980963 10154 server_base.cc:1048] running on GCE node
I20250627 13:56:03.981518 10154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:03.982470 10154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:03.983668 10154 hybrid_clock.cc:648] HybridClock initialized: now 1751032563983617 us; error 40 us; skew 500 ppm
I20250627 13:56:03.983958 10154 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:03.986280 10154 webserver.cc:469] Webserver started at http://127.8.167.193:41461/ using document root <none> and password file <none>
I20250627 13:56:03.987004 10154 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:03.987093 10154 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:03.987393 10154 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:56:03.989801 10154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/instance:
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:03.990602 10154 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal/instance:
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:03.996006 10154 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250627 13:56:03.999547 10168 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.001083 10154 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250627 13:56:04.001235 10154 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:04.001418 10154 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:04.015530 10154 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:04.016361 10154 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:04.016603 10154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:04.017659 10154 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:04.019294 10154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:04.019385 10154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.019465 10154 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:04.019526 10154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.052979 10154 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:37697
I20250627 13:56:04.053030 10280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:37697 every 8 connection(s)
I20250627 13:56:04.054466 10154 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:56:04.064882 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10154
I20250627 13:56:04.065117 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal/instance
I20250627 13:56:04.068056 10281 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:04.068459 10281 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:04.068485 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.194:0
--local_ip_for_outbound_sockets=127.8.167.194
--webserver_interface=127.8.167.194
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:56:04.069429 10281 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:04.117236 10094 ts_manager.cc:194] Registered new tserver with Master: 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:56:04.119122 10094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:58963
W20250627 13:56:04.405285 10285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:04.405674 10285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:04.405813 10285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:04.414397 10285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:04.414618 10285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.194
I20250627 13:56:04.424170 10285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.167.194
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.194
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:04.425900 10285 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:04.427735 10285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:04.439285 10291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:04.439374 10292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:04.439689 10294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:04.440246 10285 server_base.cc:1048] running on GCE node
I20250627 13:56:04.440919 10285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:04.441895 10285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:04.443115 10285 hybrid_clock.cc:648] HybridClock initialized: now 1751032564443059 us; error 68 us; skew 500 ppm
I20250627 13:56:04.443471 10285 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:04.445856 10285 webserver.cc:469] Webserver started at http://127.8.167.194:37159/ using document root <none> and password file <none>
I20250627 13:56:04.446712 10285 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:04.446813 10285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:04.447180 10285 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:56:04.449761 10285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/instance:
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.450645 10285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal/instance:
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.455997 10285 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250627 13:56:04.459312 10300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.460656 10285 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250627 13:56:04.460821 10285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.461016 10285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:04.479841 10285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:04.480744 10285 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:04.481021 10285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:04.481997 10285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:04.483640 10285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:04.483743 10285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.483863 10285 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:04.483938 10285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.522902 10285 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.194:37449
I20250627 13:56:04.522980 10412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.194:37449 every 8 connection(s)
I20250627 13:56:04.524281 10285 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
I20250627 13:56:04.526547 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10285
I20250627 13:56:04.526726 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal/instance
I20250627 13:56:04.530673 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.195:0
--local_ip_for_outbound_sockets=127.8.167.195
--webserver_interface=127.8.167.195
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:56:04.538949 10413 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:04.539355 10413 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:04.540405 10413 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:04.590011 10094 ts_manager.cc:194] Registered new tserver with Master: 9e690d05ade74ce38ef7c57ca1ff4709 (127.8.167.194:37449)
I20250627 13:56:04.591570 10094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.194:53519
W20250627 13:56:04.871083 10416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:04.871407 10416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:04.871512 10416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:04.879472 10416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:04.879714 10416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.195
I20250627 13:56:04.889446 10416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.167.195
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.195
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:04.891127 10416 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:04.892976 10416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:04.905416 10425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:04.905471 10422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:04.905344 10423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:04.906188 10416 server_base.cc:1048] running on GCE node
I20250627 13:56:04.906890 10416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:04.907872 10416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:04.909073 10416 hybrid_clock.cc:648] HybridClock initialized: now 1751032564909027 us; error 36 us; skew 500 ppm
I20250627 13:56:04.909404 10416 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:04.911720 10416 webserver.cc:469] Webserver started at http://127.8.167.195:35703/ using document root <none> and password file <none>
I20250627 13:56:04.912469 10416 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:04.912566 10416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:04.912926 10416 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:56:04.915539 10416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/instance:
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.916347 10416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal/instance:
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.921900 10416 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250627 13:56:04.925268 10431 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.926779 10416 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:56:04.926937 10416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:04.927157 10416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:04.944442 10416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:04.945290 10416 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:04.945546 10416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:04.946563 10416 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:04.948169 10416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:04.948266 10416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.948371 10416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:04.948443 10416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:04.981300 10416 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.195:34991
I20250627 13:56:04.981367 10543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.195:34991 every 8 connection(s)
I20250627 13:56:04.982860 10416 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
I20250627 13:56:04.983999 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10416
I20250627 13:56:04.984179 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal/instance
I20250627 13:56:04.988085 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.196:0
--local_ip_for_outbound_sockets=127.8.167.196
--webserver_interface=127.8.167.196
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:56:04.999032 10544 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:04.999413 10544 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:05.000427 10544 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:05.048286 10094 ts_manager.cc:194] Registered new tserver with Master: 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:05.049837 10094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.195:57819
I20250627 13:56:05.122504 10281 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:56:05.357448 10547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:05.357771 10547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:05.357990 10547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:05.366081 10547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:05.366334 10547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.196
I20250627 13:56:05.375901 10547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.8.167.196
--webserver_port=0
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.196
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:05.377557 10547 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:05.379523 10547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:05.392537 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:05.392416 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:05.392832 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:05.393882 10547 server_base.cc:1048] running on GCE node
I20250627 13:56:05.394587 10547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:05.395576 10547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:05.396816 10547 hybrid_clock.cc:648] HybridClock initialized: now 1751032565396752 us; error 61 us; skew 500 ppm
I20250627 13:56:05.397135 10547 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:05.403997 10547 webserver.cc:469] Webserver started at http://127.8.167.196:34547/ using document root <none> and password file <none>
I20250627 13:56:05.404794 10547 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:05.404912 10547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:05.405268 10547 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:56:05.407894 10547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/instance:
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:56:05.408722 10547 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal/instance:
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:56:05.414554 10547 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250627 13:56:05.418175 10562 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:05.420068 10547 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:56:05.420219 10547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:56:05.420418 10547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:05.438807 10547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:05.440106 10547 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:05.440459 10547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:05.441830 10547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:05.444308 10547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:05.444458 10547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:05.444588 10547 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:05.444669 10547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:05.480988 10547 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.196:43821
I20250627 13:56:05.481041 10674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.196:43821 every 8 connection(s)
I20250627 13:56:05.482663 10547 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
I20250627 13:56:05.490192 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10547
I20250627 13:56:05.490394 8863 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal/instance
I20250627 13:56:05.496380 10675 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:05.496749 10675 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:05.497606 10675 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:05.548657 10094 ts_manager.cc:194] Registered new tserver with Master: 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821)
I20250627 13:56:05.549602 10094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.196:41961
I20250627 13:56:05.557680 8863 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250627 13:56:05.575271 8863 test_util.cc:276] Using random seed: 168799808
I20250627 13:56:05.593664 10094 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60462:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:56:05.594731 10413 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:05.626811 10610 tablet_service.cc:1468] Processing CreateTablet for tablet 648e63fd67a54d22afac864a578a6912 (DEFAULT_TABLE table=test-workload [id=2e2a0da69e014613b3ac02fa79fadcd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:56:05.628428 10610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648e63fd67a54d22afac864a578a6912. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:05.633090 10216 tablet_service.cc:1468] Processing CreateTablet for tablet 648e63fd67a54d22afac864a578a6912 (DEFAULT_TABLE table=test-workload [id=2e2a0da69e014613b3ac02fa79fadcd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:56:05.634619 10216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648e63fd67a54d22afac864a578a6912. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:05.634152 10479 tablet_service.cc:1468] Processing CreateTablet for tablet 648e63fd67a54d22afac864a578a6912 (DEFAULT_TABLE table=test-workload [id=2e2a0da69e014613b3ac02fa79fadcd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:56:05.635632 10479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648e63fd67a54d22afac864a578a6912. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:56:05.639505 10699 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap starting.
I20250627 13:56:05.642926 10699 tablet_bootstrap.cc:654] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Neither blocks nor log segments found. Creating new log.
I20250627 13:56:05.644270 10699 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:05.647001 10699 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: No bootstrap required, opened a new log
I20250627 13:56:05.647330 10699 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20250627 13:56:05.648875 10702 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap starting.
I20250627 13:56:05.652850 10702 tablet_bootstrap.cc:654] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Neither blocks nor log segments found. Creating new log.
I20250627 13:56:05.654256 10702 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:05.655025 10701 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap starting.
I20250627 13:56:05.655319 10699 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.655969 10699 raft_consensus.cc:383] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:56:05.656167 10699 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Initialized, Role: FOLLOWER
I20250627 13:56:05.657004 10699 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.658944 10702 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: No bootstrap required, opened a new log
I20250627 13:56:05.659303 10702 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250627 13:56:05.662573 10675 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:05.662737 10701 tablet_bootstrap.cc:654] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Neither blocks nor log segments found. Creating new log.
I20250627 13:56:05.663350 10699 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20250627 13:56:05.664067 10701 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:05.666792 10701 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: No bootstrap required, opened a new log
I20250627 13:56:05.666711 10702 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.667048 10701 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20250627 13:56:05.667137 10702 raft_consensus.cc:383] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:56:05.667274 10702 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e364b586e84dba94e1678b85d63274, State: Initialized, Role: FOLLOWER
I20250627 13:56:05.667975 10702 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.669224 10544 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:05.671028 10702 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250627 13:56:05.673384 10701 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.673686 10701 raft_consensus.cc:383] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:56:05.673811 10701 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397f03f0d404d6692a8318392f3ba7c, State: Initialized, Role: FOLLOWER
I20250627 13:56:05.674436 10701 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.675740 10701 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250627 13:56:05.681016 10704 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:05.681232 10704 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.682900 10704 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:05.690126 10499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
I20250627 13:56:05.690436 10236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:56:05.690613 10499 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 0.
I20250627 13:56:05.690862 10236 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 0.
I20250627 13:56:05.691325 10566 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf; no voters:
I20250627 13:56:05.691700 10704 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:56:05.691849 10704 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:56:05.691957 10704 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:56:05.693991 10704 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.694562 10704 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 1 election: Requested vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:05.694962 10236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
I20250627 13:56:05.695071 10499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274"
I20250627 13:56:05.695122 10236 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:56:05.695250 10499 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:56:05.697232 10236 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 1.
I20250627 13:56:05.697396 10499 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 1.
I20250627 13:56:05.697647 10564 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters:
I20250627 13:56:05.697952 10704 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:56:05.698403 10704 raft_consensus.cc:695] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 LEADER]: Becoming Leader. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:56:05.698945 10704 consensus_queue.cc:237] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:05.703547 10094 catalog_manager.cc:5582] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196). New cstate: current_term: 1 leader_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } health_report { overall_health: UNKNOWN } } }
I20250627 13:56:05.709427 8863 maintenance_mode-itest.cc:745] Restarting batch of 4 tservers: 9e690d05ade74ce38ef7c57ca1ff4709,7397f03f0d404d6692a8318392f3ba7c,2cbadb2aefb24b38b5dca68b10368ddf,26e364b586e84dba94e1678b85d63274
W20250627 13:56:05.734356 10676 tablet.cc:2378] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:56:05.736215 10545 tablet.cc:2378] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:56:05.796473 10704 consensus_queue.cc:1035] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:56:05.799664 10704 consensus_queue.cc:1035] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250627 13:56:05.807367 10282 tablet.cc:2378] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:56:06.728766 10095 ts_manager.cc:295] Set tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 to MAINTENANCE_MODE
I20250627 13:56:08.570760 10095 ts_manager.cc:295] Set tserver state for 2cbadb2aefb24b38b5dca68b10368ddf to MAINTENANCE_MODE
I20250627 13:56:08.695717 10095 ts_manager.cc:295] Set tserver state for 26e364b586e84dba94e1678b85d63274 to MAINTENANCE_MODE
I20250627 13:56:08.823714 10095 ts_manager.cc:295] Set tserver state for 7397f03f0d404d6692a8318392f3ba7c to MAINTENANCE_MODE
W20250627 13:56:08.872645 10711 meta_cache.cc:1261] Time spent looking up entry by key: real 0.091s user 0.004s sys 0.000s
I20250627 13:56:11.301463 10348 tablet_service.cc:1423] Tablet server 9e690d05ade74ce38ef7c57ca1ff4709 set to quiescing
I20250627 13:56:11.301738 10348 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:13.620381 10479 tablet_service.cc:1423] Tablet server 26e364b586e84dba94e1678b85d63274 set to quiescing
I20250627 13:56:13.620790 10479 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:13.930938 10610 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:56:13.931248 10610 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:13.959353 10809 raft_consensus.cc:991] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: : Instructing follower 7397f03f0d404d6692a8318392f3ba7c to start an election
I20250627 13:56:13.964843 10730 raft_consensus.cc:1079] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 LEADER]: Signalling peer 7397f03f0d404d6692a8318392f3ba7c to start an election
I20250627 13:56:13.966225 10235 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "648e63fd67a54d22afac864a578a6912"
dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
from {username='slave'} at 127.8.167.196:52685
I20250627 13:56:13.967208 10216 tablet_service.cc:1423] Tablet server 7397f03f0d404d6692a8318392f3ba7c set to quiescing
I20250627 13:56:13.967461 10216 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:13.976332 10235 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:56:13.976523 10235 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:56:13.979677 10235 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:13.984961 10235 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 2 election: Requested vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:14.034761 10236 raft_consensus.cc:1238] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Rejecting Update request from peer 2cbadb2aefb24b38b5dca68b10368ddf for earlier term 1. Current term is 2. Ops: [1.478-1.480]
I20250627 13:56:14.035840 10730 consensus_queue.cc:1046] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: INVALID_TERM, Last received: 1.477, Next index: 478, Last known committed idx: 476, Time since last communication: 0.000s
I20250627 13:56:14.041095 10236 raft_consensus.cc:1238] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Rejecting Update request from peer 2cbadb2aefb24b38b5dca68b10368ddf for earlier term 1. Current term is 2. Ops: [1.478-1.480]
I20250627 13:56:14.041678 10498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: true dest_uuid: "26e364b586e84dba94e1678b85d63274"
I20250627 13:56:14.041980 10498 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:56:14.042284 10730 consensus_queue.cc:1046] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: INVALID_TERM, Last received: 1.477, Next index: 478, Last known committed idx: 476, Time since last communication: 0.000s
I20250627 13:56:14.046384 10498 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7397f03f0d404d6692a8318392f3ba7c for term 2 because replica has last-logged OpId of term: 1 index: 480, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 477.
I20250627 13:56:14.047788 10803 raft_consensus.cc:3053] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:56:14.047922 10803 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:56:14.048388 10803 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 480, Committed index: 480, Last appended: 1.480, Last appended by leader: 480, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:14.050386 10803 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 1 FOLLOWER]: Advancing to term 2
W20250627 13:56:14.050643 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.051469 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:14.055109 10236 raft_consensus.cc:1238] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Rejecting Update request from peer 2cbadb2aefb24b38b5dca68b10368ddf for earlier term 1. Current term is 2. Ops: [1.478-1.480]
W20250627 13:56:14.058753 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.063063 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.065968 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.068418 10727 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.021s ago.
W20250627 13:56:14.072202 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.076130 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:14.081405 10630 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: true dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
I20250627 13:56:14.081954 10630 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7397f03f0d404d6692a8318392f3ba7c for term 2 because replica has last-logged OpId of term: 1 index: 480, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 477.
W20250627 13:56:14.082222 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.082378 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:14.082640 10170 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
W20250627 13:56:14.083943 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:14.084937 10869 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250627 13:56:14.091116 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.091143 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.092921 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.098326 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.098377 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.105397 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.106949 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.110780 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.115379 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.116647 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.119858 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.124650 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.127718 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.130867 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.133257 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.141786 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.143505 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.144846 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.156639 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.157980 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.158246 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.171411 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.173477 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.174808 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.190578 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.191004 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.191184 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.208474 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.209000 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.210840 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.226464 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.227190 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.228338 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.246344 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.248662 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.249747 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.268309 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.268337 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.269542 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.290807 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.291249 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.293149 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.310870 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.315413 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.315547 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.325935 10871 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:14.333415 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.338254 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.340993 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.353487 10803 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:14.358006 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.363688 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.363782 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.384328 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.389040 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.390566 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.411243 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.417335 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.417804 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.437387 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.443674 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.443943 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.468083 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.473217 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.476032 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.499485 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.501304 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.505117 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.512115 10869 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:14.530484 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.531780 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.534276 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.561112 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.566497 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.569419 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.593943 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.601356 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.601356 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.625296 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.633729 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.633729 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.661070 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.672094 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.672596 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.701112 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.707726 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.707681 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.735236 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.742628 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.742864 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.774389 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.781400 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.783416 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.814924 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.820124 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.820415 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.852918 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.857120 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.858254 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.893993 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.897552 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.898646 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.940766 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.941982 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:14.942633 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.984148 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:14.986192 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:14.987919 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.028720 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.032353 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.034216 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.075673 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.077575 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.077575 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.123056 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.124558 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.124923 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.172454 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.172808 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.174279 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.218595 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.220885 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.222716 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.268002 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.271224 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.273131 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.321630 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.323359 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.323359 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.372299 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.372469 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.373845 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:15.399626 10479 tablet_service.cc:1423] Tablet server 26e364b586e84dba94e1678b85d63274 set to quiescing
I20250627 13:56:15.399860 10479 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:15.424165 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.424163 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.427194 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.477037 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.480491 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.481423 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.533640 10195 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.533643 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.533913 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.586653 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.586697 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.588347 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:15.614699 10610 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:56:15.614876 10610 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:15.641703 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.643362 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.645102 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:15.698436 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10285
W20250627 13:56:15.699481 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.699780 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.701802 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:15.707034 10408 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:56:15.757102 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.759040 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.759040 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:15.791023 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10285
I20250627 13:56:15.809120 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.194:37449
--local_ip_for_outbound_sockets=127.8.167.194
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=37159
--webserver_interface=127.8.167.194
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:15.820333 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.822630 10588 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.823107 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.880235 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.881695 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:15.883107 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.939178 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:15.940568 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:15.944181 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.001993 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.003676 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.005108 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.063563 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.063675 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:16.069640 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:16.123970 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:16.125418 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.133144 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.161203 10899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:16.161509 10899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:16.161607 10899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:16.169615 10899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:16.169799 10899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.194
I20250627 13:56:16.179456 10899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.194:37449
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.167.194
--webserver_port=37159
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.194
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:16.181149 10899 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:16.183056 10899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:16.186852 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.190256 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.196113 10904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:16.196146 10905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:16.196231 10907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:16.197191 10899 server_base.cc:1048] running on GCE node
I20250627 13:56:16.197799 10899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:16.199033 10899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250627 13:56:16.199642 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:16.200271 10899 hybrid_clock.cc:648] HybridClock initialized: now 1751032576200211 us; error 49 us; skew 500 ppm
I20250627 13:56:16.200553 10899 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:16.203011 10899 webserver.cc:469] Webserver started at http://127.8.167.194:37159/ using document root <none> and password file <none>
I20250627 13:56:16.203769 10899 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:16.203876 10899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:16.209363 10899 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:56:16.212147 10913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:16.213500 10899 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:56:16.213629 10899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:16.214344 10899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:16.237636 10899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:16.238571 10899 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:16.238842 10899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:16.239858 10899 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:16.241485 10899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:16.241562 10899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:16.241654 10899 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:16.241704 10899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250627 13:56:16.255513 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.257151 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:16.267232 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:16.276786 10899 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.194:37449
I20250627 13:56:16.276825 11025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.194:37449 every 8 connection(s)
I20250627 13:56:16.278398 10899 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
I20250627 13:56:16.282627 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 10899
I20250627 13:56:16.282882 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10154
I20250627 13:56:16.289831 10276 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:16.292982 11026 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:16.293349 11026 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:16.294317 11026 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:16.296054 10095 ts_manager.cc:194] Re-registered known tserver with Master: 9e690d05ade74ce38ef7c57ca1ff4709 (127.8.167.194:37449)
I20250627 13:56:16.296916 10095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.194:55713
W20250627 13:56:16.325440 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.334544 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.395776 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.402861 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.412324 10196 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48968: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:16.412613 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10154
I20250627 13:56:16.437211 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:37697
--local_ip_for_outbound_sockets=127.8.167.193
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=41461
--webserver_interface=127.8.167.193
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:56:16.456667 10714 meta_cache.cc:1510] marking tablet server 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697) as failed
W20250627 13:56:16.457041 10714 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.8.167.193:37697: connect: Connection refused (error 111)
W20250627 13:56:16.462153 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.472545 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.479048 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.533502 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.550238 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.615913 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:16.625443 10716 meta_cache.cc:1510] marking tablet server 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697) as failed
W20250627 13:56:16.676354 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.686606 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.687424 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.753289 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.762898 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:16.792300 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:16.792654 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:16.792793 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:16.800791 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:16.801024 11033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:56:16.810843 11033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=41461
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:16.812489 11033 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:16.814370 11033 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:16.826844 11042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:16.826766 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:16.826778 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:16.827855 11033 server_base.cc:1048] running on GCE node
I20250627 13:56:16.828464 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:16.829612 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:16.830839 11033 hybrid_clock.cc:648] HybridClock initialized: now 1751032576830777 us; error 49 us; skew 500 ppm
I20250627 13:56:16.831205 11033 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250627 13:56:16.832052 10590 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35638: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:16.839743 11033 webserver.cc:469] Webserver started at http://127.8.167.193:41461/ using document root <none> and password file <none>
I20250627 13:56:16.840626 11033 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:16.840740 11033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:16.846331 11033 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:56:16.849084 11048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:16.850567 11033 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:56:16.850754 11033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:16.851441 11033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:16.879827 11033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:16.880717 11033 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:16.881007 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:16.882014 11033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:16.884263 11055 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:16.889983 11033 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:16.890131 11033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20250627 13:56:16.890288 11033 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:16.893527 11033 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:16.893625 11033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250627 13:56:16.894107 11055 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap starting.
W20250627 13:56:16.899240 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.903255 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:16.905562 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:16.941776 11033 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:37697
I20250627 13:56:16.941826 11161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:37697 every 8 connection(s)
I20250627 13:56:16.944144 11033 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:56:16.950186 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 11033
I20250627 13:56:16.950402 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10547
I20250627 13:56:16.975291 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:16.979745 11162 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:16.980270 11162 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:16.981563 11162 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:16.987744 10095 ts_manager.cc:194] Re-registered known tserver with Master: 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:56:16.989499 10095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:59165
I20250627 13:56:17.081651 11055 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:17.108620 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10547
I20250627 13:56:17.133359 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.196:43821
--local_ip_for_outbound_sockets=127.8.167.196
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=34547
--webserver_interface=127.8.167.196
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:17.279590 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:17.290097 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:17.290103 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:17.299903 11026 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:56:17.486913 11171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:17.487247 11171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:17.487378 11171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:17.495797 11171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:17.496014 11171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.196
I20250627 13:56:17.505964 11171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.196:43821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.8.167.196
--webserver_port=34547
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.196
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:17.508102 11171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:17.510448 11171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:17.515033 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:17.526799 11177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:17.527079 11179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:17.527240 11176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:17.528431 11171 server_base.cc:1048] running on GCE node
I20250627 13:56:17.529083 11171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:17.530331 11171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250627 13:56:17.531476 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:17.532152 11171 hybrid_clock.cc:648] HybridClock initialized: now 1751032577532084 us; error 47 us; skew 500 ppm
I20250627 13:56:17.532485 11171 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250627 13:56:17.532635 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:17.535180 11171 webserver.cc:469] Webserver started at http://127.8.167.196:34547/ using document root <none> and password file <none>
I20250627 13:56:17.535960 11171 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:17.536094 11171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:17.541684 11171 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20250627 13:56:17.544881 11185 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:17.546294 11171 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:56:17.546444 11171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:56:17.547152 11171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:17.576294 11171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:17.577266 11171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:17.577525 11171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:17.578568 11171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:17.580811 11192 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:17.586287 11171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:17.586370 11171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250627 13:56:17.586488 11171 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:17.589391 11171 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:17.589506 11171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250627 13:56:17.590003 11192 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap starting.
W20250627 13:56:17.596027 10685 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.196:43821: connect: Connection refused (error 111)
I20250627 13:56:17.644317 11171 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.196:43821
I20250627 13:56:17.644361 11298 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.196:43821 every 8 connection(s)
I20250627 13:56:17.646054 11171 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
I20250627 13:56:17.648101 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 11171
I20250627 13:56:17.648315 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10416
I20250627 13:56:17.689756 10539 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:17.700731 11299 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:17.701150 11299 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:17.702118 11299 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:17.705232 10095 ts_manager.cc:194] Re-registered known tserver with Master: 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821)
I20250627 13:56:17.706772 10095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.196:34841
I20250627 13:56:17.861388 11192 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Log is configured to *not* fsync() on all Append() calls
W20250627 13:56:17.936000 10455 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:17.940558 10459 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:17.941371 10457 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48710: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:17.942832 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10416
I20250627 13:56:17.993361 11162 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:56:18.006227 10687 connection.cc:537] client connection to 127.8.167.195:34991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:56:18.005846 10081 connection.cc:537] server connection from 127.8.167.195:57819 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:56:18.006554 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.195:34991
--local_ip_for_outbound_sockets=127.8.167.195
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=35703
--webserver_interface=127.8.167.195
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:56:18.019735 10715 meta_cache.cc:1510] marking tablet server 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) as failed
I20250627 13:56:18.059091 10714 meta_cache.cc:1510] marking tablet server 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) as failed
I20250627 13:56:18.227531 10716 meta_cache.cc:1510] marking tablet server 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) as failed
I20250627 13:56:18.350556 11055 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=477 overwritten=0 applied=476 ignored=0} inserts{seen=23750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:56:18.351251 11055 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap complete.
I20250627 13:56:18.352711 11055 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent bootstrapping tablet: real 1.459s user 1.317s sys 0.128s
I20250627 13:56:18.357738 11055 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:18.360611 11055 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397f03f0d404d6692a8318392f3ba7c, State: Initialized, Role: FOLLOWER
I20250627 13:56:18.361505 11055 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 476, Last appended: 1.477, Last appended by leader: 477, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:18.364805 11055 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20250627 13:56:18.394085 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:18.483620 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:18.495713 11308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:18.496162 11308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:18.496337 11308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:18.509470 11308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:18.509756 11308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.195
I20250627 13:56:18.525357 11308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.195:34991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.167.195
--webserver_port=35703
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.195
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:18.527571 11308 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:18.530187 11308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:18.545975 11315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:18.546115 11316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:18.547469 11318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:18.548426 11308 server_base.cc:1048] running on GCE node
I20250627 13:56:18.549108 11308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:18.550292 11308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:18.551524 11308 hybrid_clock.cc:648] HybridClock initialized: now 1751032578551460 us; error 54 us; skew 500 ppm
I20250627 13:56:18.551858 11308 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:18.554502 11308 webserver.cc:469] Webserver started at http://127.8.167.195:35703/ using document root <none> and password file <none>
I20250627 13:56:18.555320 11308 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:18.555434 11308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:18.561307 11308 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:56:18.564404 11324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:18.565886 11308 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:56:18.566066 11308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:18.566778 11308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250627 13:56:18.578177 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:18.584061 11308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:18.584964 11308 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:18.585239 11308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:18.586345 11308 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:18.588933 11331 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:18.594496 11308 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:18.594578 11308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20250627 13:56:18.594697 11308 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:18.597604 11308 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:18.597721 11308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250627 13:56:18.598249 11331 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap starting.
W20250627 13:56:18.638808 10687 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
I20250627 13:56:18.660231 11310 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:18.660531 11310 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:18.662705 11310 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
W20250627 13:56:18.668011 11052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
W20250627 13:56:18.668474 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:18.691352 11052 leader_election.cc:336] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
I20250627 13:56:18.689853 11254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
W20250627 13:56:18.693378 11050 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:18.693591 11050 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
I20250627 13:56:18.694363 11310 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:56:18.695361 11308 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.195:34991
I20250627 13:56:18.695469 11441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.195:34991 every 8 connection(s)
I20250627 13:56:18.697508 11308 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
I20250627 13:56:18.698320 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 11308
I20250627 13:56:18.710158 11299 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:18.759884 11442 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:18.760342 11442 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:18.761538 11442 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:18.764926 10095 ts_manager.cc:194] Re-registered known tserver with Master: 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:18.766503 10095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.195:34405
I20250627 13:56:18.871110 11331 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Log is configured to *not* fsync() on all Append() calls
W20250627 13:56:18.872675 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:18.963661 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:18.969796 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:19.029913 11310 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:19.030146 11310 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.030846 11310 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:19.031611 11254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
W20250627 13:56:19.034718 11050 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:19.041406 11393 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
W20250627 13:56:19.044879 11052 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:19.045038 11052 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
I20250627 13:56:19.045480 11310 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250627 13:56:19.164844 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.266438 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.269991 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:19.362699 11373 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:19.415383 10961 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:19.423131 11234 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:19.445765 11097 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:19.477689 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:19.536773 11192 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replayed 1/1 log segments. Stats: ops{read=480 overwritten=0 applied=480 ignored=0} inserts{seen=23950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:56:19.537484 11192 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap complete.
I20250627 13:56:19.539270 11192 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent bootstrapping tablet: real 1.950s user 1.684s sys 0.179s
I20250627 13:56:19.544848 11192 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.546073 11192 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Initialized, Role: FOLLOWER
I20250627 13:56:19.546792 11192 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 480, Last appended: 1.480, Last appended by leader: 480, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.548839 11192 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
W20250627 13:56:19.565907 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.571907 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.584748 11214 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:19.586947 11481 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:19.587148 11481 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.587904 11481 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:19.588265 11393 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
I20250627 13:56:19.588451 11254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 477 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
I20250627 13:56:19.588908 11254 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7397f03f0d404d6692a8318392f3ba7c for term 3 because replica has last-logged OpId of term: 1 index: 480, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 477.
W20250627 13:56:19.588948 11052 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:19.589558 11050 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
I20250627 13:56:19.589818 11481 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250627 13:56:19.770663 11442 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:56:19.770606 11214 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:19.781574 11214 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:19.793632 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.870855 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.884908 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:19.899798 11214 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:19.906720 11480 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:19.907032 11480 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.908697 11480 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:19.916715 11117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 3 candidate_status { last_received { term: 1 index: 480 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:56:19.917114 11117 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 2.
I20250627 13:56:19.917362 11393 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 3 candidate_status { last_received { term: 1 index: 480 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
I20250627 13:56:19.917706 11187 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters:
W20250627 13:56:19.918025 11189 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:19.918202 11480 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250627 13:56:19.918344 11480 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:56:19.918445 11480 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:56:19.921789 11480 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.922387 11480 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 election: Requested vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:19.922868 11393 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 3 candidate_status { last_received { term: 1 index: 480 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274"
I20250627 13:56:19.922876 11117 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 3 candidate_status { last_received { term: 1 index: 480 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
I20250627 13:56:19.923060 11117 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 2 FOLLOWER]: Advancing to term 3
W20250627 13:56:19.923300 11189 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:19.927804 11117 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 3.
I20250627 13:56:19.928247 11187 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274
I20250627 13:56:19.928717 11480 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:56:19.929750 11480 raft_consensus.cc:695] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 LEADER]: Becoming Leader. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:56:19.930356 11480 consensus_queue.cc:237] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 480, Committed index: 480, Last appended: 1.480, Last appended by leader: 480, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:19.935971 10092 catalog_manager.cc:5582] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } health_report { overall_health: UNKNOWN } } }
W20250627 13:56:19.943976 10715 scanner-internal.cc:458] Time spent opening tablet: real 5.887s user 0.007s sys 0.004s
I20250627 13:56:20.045336 11117 raft_consensus.cc:1273] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 3 FOLLOWER]: Refusing update from remote peer 2cbadb2aefb24b38b5dca68b10368ddf: Log matching property violated. Preceding OpId in replica: term: 1 index: 477. Preceding OpId from leader: term: 3 index: 481. (index mismatch)
I20250627 13:56:20.046612 11480 consensus_queue.cc:1035] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 481, Last known committed idx: 476, Time since last communication: 0.000s
W20250627 13:56:20.057885 11189 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250627 13:56:20.072556 10714 scanner-internal.cc:458] Time spent opening tablet: real 6.086s user 0.014s sys 0.008s
W20250627 13:56:20.112978 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:20.158571 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:20.159704 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:20.239560 10716 scanner-internal.cc:458] Time spent opening tablet: real 6.035s user 0.005s sys 0.004s
W20250627 13:56:20.575917 11189 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20250627 13:56:20.584506 11331 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replayed 1/1 log segments. Stats: ops{read=480 overwritten=0 applied=480 ignored=0} inserts{seen=23950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:56:20.585335 11331 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap complete.
I20250627 13:56:20.587251 11331 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent bootstrapping tablet: real 1.989s user 1.829s sys 0.087s
I20250627 13:56:20.592839 11331 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:20.593822 11331 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e364b586e84dba94e1678b85d63274, State: Initialized, Role: FOLLOWER
I20250627 13:56:20.594729 11331 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 480, Last appended: 1.480, Last appended by leader: 480, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:20.603420 11331 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20250627 13:56:20.706759 11393 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:56:20.832540 11503 mvcc.cc:204] Tried to move back new op lower bound from 7172229450416164864 to 7172229447399129088. Current Snapshot: MvccSnapshot[applied={T|T < 7172229448478453760}]
I20250627 13:56:25.908063 11097 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:25.919363 11373 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:25.941769 10961 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:25.986294 11234 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250627 13:56:26.931826 10711 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.002s sys 0.000s
W20250627 13:56:27.043358 10685 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.8.167.196:43821, user_credentials={real_user=slave}} blocked reactor thread for 131787us
I20250627 13:56:27.308974 10092 ts_manager.cc:284] Unset tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 from MAINTENANCE_MODE
I20250627 13:56:27.310117 11162 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:27.325096 11026 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:27.876777 11299 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:27.910591 11442 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:27.973877 10095 ts_manager.cc:284] Unset tserver state for 2cbadb2aefb24b38b5dca68b10368ddf from MAINTENANCE_MODE
I20250627 13:56:27.983987 10092 ts_manager.cc:284] Unset tserver state for 7397f03f0d404d6692a8318392f3ba7c from MAINTENANCE_MODE
I20250627 13:56:28.317031 11162 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:28.331689 10086 ts_manager.cc:284] Unset tserver state for 26e364b586e84dba94e1678b85d63274 from MAINTENANCE_MODE
I20250627 13:56:28.333515 11026 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:28.883090 11299 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:28.979986 11442 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:29.324910 11162 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:32.780814 10092 ts_manager.cc:295] Set tserver state for 7397f03f0d404d6692a8318392f3ba7c to MAINTENANCE_MODE
I20250627 13:56:33.090106 10092 ts_manager.cc:295] Set tserver state for 2cbadb2aefb24b38b5dca68b10368ddf to MAINTENANCE_MODE
I20250627 13:56:34.119866 10092 ts_manager.cc:295] Set tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 to MAINTENANCE_MODE
I20250627 13:56:34.904611 10092 ts_manager.cc:295] Set tserver state for 26e364b586e84dba94e1678b85d63274 to MAINTENANCE_MODE
I20250627 13:56:36.368402 11097 tablet_service.cc:1423] Tablet server 7397f03f0d404d6692a8318392f3ba7c set to quiescing
I20250627 13:56:36.368803 11097 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:36.697258 11189 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.195:34991, user_credentials={real_user=slave}} blocked reactor thread for 62503.8us
I20250627 13:56:39.448153 10961 tablet_service.cc:1423] Tablet server 9e690d05ade74ce38ef7c57ca1ff4709 set to quiescing
I20250627 13:56:39.448453 10961 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:42.050841 11234 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:56:42.051100 11234 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:42.064714 11754 raft_consensus.cc:991] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: : Instructing follower 26e364b586e84dba94e1678b85d63274 to start an election
I20250627 13:56:42.064881 11754 raft_consensus.cc:1079] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 LEADER]: Signalling peer 26e364b586e84dba94e1678b85d63274 to start an election
I20250627 13:56:42.066965 11393 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "648e63fd67a54d22afac864a578a6912"
dest_uuid: "26e364b586e84dba94e1678b85d63274"
from {username='slave'} at 127.8.167.196:46011
I20250627 13:56:42.067234 11393 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:56:42.067430 11393 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:56:42.071853 11393 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:42.073638 11393 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 4 election: Requested vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:56:42.104323 11393 raft_consensus.cc:1238] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Rejecting Update request from peer 2cbadb2aefb24b38b5dca68b10368ddf for earlier term 3. Current term is 4. Ops: [3.1954-3.1954]
I20250627 13:56:42.105737 11753 consensus_queue.cc:1046] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 }, Status: INVALID_TERM, Last received: 3.1953, Next index: 1954, Last known committed idx: 1953, Time since last communication: 0.000s
I20250627 13:56:42.106161 11753 raft_consensus.cc:3053] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 LEADER]: Stepping down as leader of term 3
I20250627 13:56:42.106257 11753 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:56:42.106585 11753 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1954, Committed index: 1954, Last appended: 3.1955, Last appended by leader: 1955, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:42.107149 11753 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:56:42.109912 11254 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 4 candidate_status { last_received { term: 3 index: 1953 } } ignore_live_leader: true dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
W20250627 13:56:42.110294 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.111366 11492 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.028s ago.
W20250627 13:56:42.115399 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:42.116042 11254 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 4 because replica has last-logged OpId of term: 3 index: 1955, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1953.
W20250627 13:56:42.116480 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:42.124977 11115 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 4 candidate_status { last_received { term: 3 index: 1953 } } ignore_live_leader: true dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
I20250627 13:56:42.125221 11115 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 3 FOLLOWER]: Advancing to term 4
W20250627 13:56:42.126096 11353 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.127463 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:42.128440 11115 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 4 because replica has last-logged OpId of term: 3 index: 1955, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 1953.
I20250627 13:56:42.129676 11326 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e364b586e84dba94e1678b85d63274; no voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c
W20250627 13:56:42.138005 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.139600 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:42.140091 11780 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250627 13:56:42.146759 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.148062 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.158083 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.158932 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.175557 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.178670 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.185770 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.189441 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.196586 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.201820 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:42.203897 11373 tablet_service.cc:1423] Tablet server 26e364b586e84dba94e1678b85d63274 set to quiescing
I20250627 13:56:42.204164 11373 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250627 13:56:42.211594 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.219671 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.223491 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.232964 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.238622 11353 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.249810 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.261260 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.271355 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.277464 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.289511 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.300098 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.310478 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.322737 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.334026 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.343122 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.352609 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.366693 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.373287 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.394434 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.395694 11753 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:42.398483 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.415427 11786 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:42.415659 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.420853 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.439983 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.446018 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.469347 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.473204 11780 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:56:42.473910 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.494362 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.503021 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.522645 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.528080 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.551558 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.557583 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.580042 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.588088 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.610936 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.620169 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.646168 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.654384 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.678666 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.686760 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.715004 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.719681 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.753334 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.755522 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.789790 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.790941 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.826782 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.827046 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.865944 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.868062 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.905444 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.907366 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:42.945012 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.951251 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:42.993148 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:42.996030 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.036095 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.039158 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.080193 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.082106 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.125895 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.125895 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.171007 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.171531 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.216454 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.219141 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.273671 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.274679 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.322562 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.324824 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.370797 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.372817 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.422083 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.425006 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.474324 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.474398 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.524967 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.525956 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.579389 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.580492 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.634711 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.634779 11076 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:43.654516 11234 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:56:43.654870 11234 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:43.690325 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.691286 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.748672 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.749047 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.803849 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.804877 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.863957 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.863962 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:43.922127 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.925760 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:43.981621 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:43.982470 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.043488 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.044482 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.105396 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.108323 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.167471 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.171439 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.232790 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.235920 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.297128 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.301604 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.362964 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.364435 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.434196 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.434820 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.505537 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.505913 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.573151 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.574895 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.643229 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.644403 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.714768 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.716375 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.787530 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:44.788795 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:44.812654 11373 tablet_service.cc:1423] Tablet server 26e364b586e84dba94e1678b85d63274 set to quiescing
I20250627 13:56:44.812870 11373 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:44.859902 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:44.860941 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:44.901386 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10899
I20250627 13:56:44.908746 11020 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:56:44.933859 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:44.936836 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:44.999871 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10899
W20250627 13:56:45.009575 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:45.010807 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:45.019491 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.194:37449
--local_ip_for_outbound_sockets=127.8.167.194
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=37159
--webserver_interface=127.8.167.194
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:45.085388 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.085419 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.165359 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:45.165649 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:45.241890 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:45.244721 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:45.320883 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.322978 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.374933 11814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:45.375262 11814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:45.375394 11814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:45.383672 11814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:45.383893 11814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.194
I20250627 13:56:45.393486 11814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.194:37449
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.167.194
--webserver_port=37159
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.194
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:45.395162 11814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:45.396946 11814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:45.403191 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:45.405977 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:45.412248 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:45.413051 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:45.413045 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:45.414155 11814 server_base.cc:1048] running on GCE node
I20250627 13:56:45.414860 11814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:45.415818 11814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:45.417042 11814 hybrid_clock.cc:648] HybridClock initialized: now 1751032605416993 us; error 44 us; skew 500 ppm
I20250627 13:56:45.417353 11814 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:45.419639 11814 webserver.cc:469] Webserver started at http://127.8.167.194:37159/ using document root <none> and password file <none>
I20250627 13:56:45.420409 11814 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:45.420526 11814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:45.426138 11814 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20250627 13:56:45.428921 11828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:45.430323 11814 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250627 13:56:45.430487 11814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:45.431205 11814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:45.446836 11814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:45.447718 11814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:45.447984 11814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:45.448951 11814 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:45.450666 11814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:45.450764 11814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:45.450899 11814 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:45.450958 11814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250627 13:56:45.482725 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:45.484237 11814 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.194:37449
I20250627 13:56:45.484285 11940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.194:37449 every 8 connection(s)
I20250627 13:56:45.485817 11814 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
W20250627 13:56:45.486687 11077 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:49072: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:45.495244 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 11814
I20250627 13:56:45.495481 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 11033
I20250627 13:56:45.500748 11941 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:45.501127 11941 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:45.502322 11941 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:45.503448 11156 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:45.504307 10086 ts_manager.cc:194] Re-registered known tserver with Master: 9e690d05ade74ce38ef7c57ca1ff4709 (127.8.167.194:37449)
I20250627 13:56:45.505659 10086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.194:35449
W20250627 13:56:45.566092 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.566114 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.650744 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:45.652462 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:45.720688 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 11033
W20250627 13:56:45.761057 10685 connection.cc:537] client connection to 127.8.167.193:37697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:56:45.761242 10685 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:56:45.761734 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:37697
--local_ip_for_outbound_sockets=127.8.167.193
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=41461
--webserver_interface=127.8.167.193
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:45.844612 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.846522 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:45.932792 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:46.019564 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.019902 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:46.108168 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.117750 11948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:46.118144 11948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:46.118258 11948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:46.126252 11948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:46.126448 11948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:56:46.136420 11948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=41461
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:46.138140 11948 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:46.139954 11948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:46.151690 11954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:46.151750 11957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:46.151628 11955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:46.152532 11948 server_base.cc:1048] running on GCE node
I20250627 13:56:46.153158 11948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:46.154122 11948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:46.155339 11948 hybrid_clock.cc:648] HybridClock initialized: now 1751032606155269 us; error 67 us; skew 500 ppm
I20250627 13:56:46.155655 11948 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:46.158007 11948 webserver.cc:469] Webserver started at http://127.8.167.193:41461/ using document root <none> and password file <none>
I20250627 13:56:46.158840 11948 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:46.158950 11948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:46.164353 11948 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:56:46.167155 11963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:46.168442 11948 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:56:46.168606 11948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:56:46.169309 11948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:46.187932 11948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:46.188789 11948 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:46.189031 11948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:46.189970 11948 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:46.192004 11970 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250627 13:56:46.197171 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:46.198287 11948 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:46.198405 11948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20250627 13:56:46.198568 11948 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:46.201736 11948 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:46.201910 11948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250627 13:56:46.202392 11970 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap starting.
I20250627 13:56:46.245230 11948 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:37697
I20250627 13:56:46.245275 12076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:37697 every 8 connection(s)
I20250627 13:56:46.247071 11948 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:56:46.257174 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 11948
I20250627 13:56:46.257396 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 11171
I20250627 13:56:46.262375 12077 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:46.262858 12077 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:46.264011 12077 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:46.267231 10086 ts_manager.cc:194] Re-registered known tserver with Master: 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:56:46.268108 11294 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:46.269201 10086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:41739
W20250627 13:56:46.285876 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.507056 11212 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:46.507321 11210 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34452: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:46.507529 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 11171
I20250627 13:56:46.509351 11941 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:46.554399 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.196:43821
--local_ip_for_outbound_sockets=127.8.167.196
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=34547
--webserver_interface=127.8.167.196
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:46.576738 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:46.584692 10715 meta_cache.cc:1510] marking tablet server 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821) as failed
W20250627 13:56:46.610239 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.645900 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.691613 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.694546 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.736868 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.751693 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.775976 10685 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.196:43821: connect: Connection refused (error 111)
I20250627 13:56:46.786484 11970 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Log is configured to *not* fsync() on all Append() calls
W20250627 13:56:46.820896 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.892638 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.948997 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:46.949337 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:46.949486 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:46.957561 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:46.957803 12085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.196
I20250627 13:56:46.967767 12085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.196:43821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.8.167.196
--webserver_port=34547
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.196
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:46.969575 12085 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:46.971876 12085 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:46.974929 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.979308 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:46.986541 12092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:46.986588 12091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:46.987371 12094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:46.987828 12085 server_base.cc:1048] running on GCE node
I20250627 13:56:46.988463 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:46.989619 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:46.990891 12085 hybrid_clock.cc:648] HybridClock initialized: now 1751032606990821 us; error 49 us; skew 500 ppm
I20250627 13:56:46.991254 12085 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:46.994078 12085 webserver.cc:469] Webserver started at http://127.8.167.196:34547/ using document root <none> and password file <none>
I20250627 13:56:46.994879 12085 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:46.994994 12085 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:47.000931 12085 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:56:47.004112 12100 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:47.005692 12085 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:56:47.005867 12085 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:56:47.006712 12085 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:47.012285 10716 meta_cache.cc:1510] marking tablet server 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821) as failed
I20250627 13:56:47.025409 12085 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:47.026448 12085 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:47.026715 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:47.027746 12085 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:47.029963 12107 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:47.035703 12085 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:47.035799 12085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250627 13:56:47.035934 12085 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:47.039009 12085 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:47.039142 12085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20250627 13:56:47.039608 12107 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap starting.
W20250627 13:56:47.062976 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:47.093950 12085 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.196:43821
I20250627 13:56:47.094137 12213 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.196:43821 every 8 connection(s)
I20250627 13:56:47.095980 12085 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
I20250627 13:56:47.105619 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 12085
I20250627 13:56:47.105875 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 11308
I20250627 13:56:47.140888 11432 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:47.156332 12214 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:47.156970 12214 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:47.158504 12214 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:47.162283 10086 ts_manager.cc:194] Re-registered known tserver with Master: 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821)
I20250627 13:56:47.164460 10086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.196:41505
I20250627 13:56:47.273264 12077 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:56:47.389200 11349 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:47.389626 11351 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:56:47.389200 11350 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51426: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:56:47.390007 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 11308
I20250627 13:56:47.436570 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.195:34991
--local_ip_for_outbound_sockets=127.8.167.195
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=35703
--webserver_interface=127.8.167.195
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:47.811573 12222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:47.811908 12222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:47.812073 12222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:47.820418 12222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:47.820679 12222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.195
W20250627 13:56:47.830737 10687 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
I20250627 13:56:47.831682 12222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.195:34991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.167.195
--webserver_port=35703
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.195
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:47.833379 12222 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:47.835294 12222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:47.857005 12227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:47.859802 12228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:47.863411 12231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:47.864027 12222 server_base.cc:1048] running on GCE node
I20250627 13:56:47.864686 12222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:47.865756 12222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:47.867071 12222 hybrid_clock.cc:648] HybridClock initialized: now 1751032607867006 us; error 59 us; skew 500 ppm
I20250627 13:56:47.867394 12222 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:47.870165 12222 webserver.cc:469] Webserver started at http://127.8.167.195:35703/ using document root <none> and password file <none>
I20250627 13:56:47.870945 12222 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:47.871058 12222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:47.876932 12222 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:56:47.879943 12236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:47.881333 12222 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20250627 13:56:47.881498 12222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:56:47.882258 12222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:47.904425 12222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:47.905345 12222 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:47.905639 12222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:47.906795 12222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:47.909154 12243 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:47.915340 12222 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:47.915452 12222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20250627 13:56:47.915616 12222 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:47.918722 12222 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:47.918915 12222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.004s sys 0.000s
I20250627 13:56:47.919322 12243 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap starting.
I20250627 13:56:47.932214 12107 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:48.169164 12214 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:48.589217 10715 meta_cache.cc:1510] marking tablet server 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) as failed
I20250627 13:56:48.650379 10714 meta_cache.cc:1510] marking tablet server 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) as failed
I20250627 13:56:48.701191 12350 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.195:34991 every 8 connection(s)
I20250627 13:56:48.701624 12222 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.195:34991
I20250627 13:56:48.703910 12222 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
I20250627 13:56:48.708102 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 12222
I20250627 13:56:48.770604 12351 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:56:48.771071 12351 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:48.772413 12351 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:56:48.775951 10086 ts_manager.cc:194] Re-registered known tserver with Master: 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:48.777596 10086 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.195:51497
I20250627 13:56:48.844188 12243 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:49.743553 12286 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:49.755798 12149 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:49.781638 12351 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:56:49.790827 11876 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:49.802331 12012 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:51.790431 11970 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replaying log segment 1/1 (2.19M/3.33M this segment, stats: ops{read=1289 overwritten=0 applied=1289 ignored=0} inserts{seen=64350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:56:52.958508 12107 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replaying log segment 1/1 (2.66M/3.38M this segment, stats: ops{read=1538 overwritten=0 applied=1537 ignored=0} inserts{seen=76750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:56:53.660095 10714 scanner-internal.cc:172] Scan of tablet 648e63fd67a54d22afac864a578a6912 at 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) deadline exceeded after 10.000s
W20250627 13:56:53.660436 10714 scanner-internal.cc:458] Time spent opening tablet: real 10.935s user 0.012s sys 0.000s
I20250627 13:56:53.812301 10715 scanner-internal.cc:172] Scan of tablet 648e63fd67a54d22afac864a578a6912 at 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) deadline exceeded after 10.000s
W20250627 13:56:53.812498 10715 scanner-internal.cc:458] Time spent opening tablet: real 11.268s user 0.014s sys 0.000s
I20250627 13:56:53.857499 12243 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replaying log segment 1/1 (2.08M/3.30M this segment, stats: ops{read=1236 overwritten=0 applied=1236 ignored=0} inserts{seen=61700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:56:54.048218 12107 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replayed 1/1 log segments. Stats: ops{read=1955 overwritten=0 applied=1954 ignored=0} inserts{seen=97600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:56:54.048903 12107 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap complete.
I20250627 13:56:54.050542 12107 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent bootstrapping tablet: real 7.011s user 6.228s sys 0.437s
I20250627 13:56:54.055495 12107 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.058269 12107 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Initialized, Role: FOLLOWER
I20250627 13:56:54.059151 12107 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1954, Last appended: 3.1955, Last appended by leader: 1955, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.061451 12107 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20250627 13:56:54.163038 12129 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:54.226068 10716 scanner-internal.cc:172] Scan of tablet 648e63fd67a54d22afac864a578a6912 at 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) deadline exceeded after 10.000s
W20250627 13:56:54.226253 10716 scanner-internal.cc:458] Time spent opening tablet: real 11.242s user 0.009s sys 0.004s
W20250627 13:56:54.250310 12129 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:54.301712 12392 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:54.302943 12392 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.305009 12392 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:54.325304 12032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
W20250627 13:56:54.329074 12102 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:54.331120 12306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
W20250627 13:56:54.334434 12104 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:54.334607 12104 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf; no voters: 26e364b586e84dba94e1678b85d63274, 7397f03f0d404d6692a8318392f3ba7c
I20250627 13:56:54.335054 12392 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:56:54.369393 11970 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1955 overwritten=0 applied=1954 ignored=0} inserts{seen=97600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:56:54.370116 11970 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap complete.
I20250627 13:56:54.371714 11970 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent bootstrapping tablet: real 8.170s user 7.650s sys 0.451s
I20250627 13:56:54.375656 11970 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.378365 11970 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397f03f0d404d6692a8318392f3ba7c, State: Initialized, Role: FOLLOWER
I20250627 13:56:54.379110 11970 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1954, Last appended: 3.1955, Last appended by leader: 1955, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.380726 11970 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
W20250627 13:56:54.403038 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:54.622777 12129 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:56:54.714465 12129 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:56:54.734498 12398 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:54.734778 12398 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.736430 12398 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:54.737272 12392 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:56:54.737445 12392 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.738322 12392 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:54.739086 12306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
W20250627 13:56:54.739779 12104 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:54.744089 12032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:56:54.744400 12032 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 4.
I20250627 13:56:54.744995 12102 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274
I20250627 13:56:54.745450 12392 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250627 13:56:54.745594 12392 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:56:54.745715 12392 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 4 FOLLOWER]: Advancing to term 5
I20250627 13:56:54.751384 12392 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.752730 12392 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 election: Requested vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:56:54.752794 12306 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
I20250627 13:56:54.752929 12169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
I20250627 13:56:54.753436 12169 raft_consensus.cc:2391] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7397f03f0d404d6692a8318392f3ba7c in current term 5: Already voted for candidate 2cbadb2aefb24b38b5dca68b10368ddf in this term.
I20250627 13:56:54.753343 12305 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274"
W20250627 13:56:54.754007 12104 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:54.754108 12032 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 5 candidate_status { last_received { term: 3 index: 1955 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
W20250627 13:56:54.754057 11967 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 5 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:56:54.754308 12032 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 4 FOLLOWER]: Advancing to term 5
I20250627 13:56:54.754652 11965 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
I20250627 13:56:54.758006 12032 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 5.
I20250627 13:56:54.758219 12398 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:56:54.758571 12102 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274
I20250627 13:56:54.758939 12392 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 FOLLOWER]: Leader election won for term 5
I20250627 13:56:54.759382 12392 raft_consensus.cc:695] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 LEADER]: Becoming Leader. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:56:54.759960 12392 consensus_queue.cc:237] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1954, Committed index: 1954, Last appended: 3.1955, Last appended by leader: 1955, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:54.764657 10086 catalog_manager.cc:5582] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } health_report { overall_health: UNKNOWN } } }
W20250627 13:56:54.778789 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:54.855644 12104 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:56:54.869962 12032 raft_consensus.cc:1273] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 5 FOLLOWER]: Refusing update from remote peer 2cbadb2aefb24b38b5dca68b10368ddf: Log matching property violated. Preceding OpId in replica: term: 3 index: 1955. Preceding OpId from leader: term: 5 index: 1956. (index mismatch)
W20250627 13:56:54.870865 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:56:54.870975 12392 consensus_queue.cc:1035] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1956, Last known committed idx: 1954, Time since last communication: 0.000s
W20250627 13:56:54.886495 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:56:55.372444 12104 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:55.919665 12104 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20250627 13:56:56.113245 12286 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:56.145968 11876 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:56.149331 12149 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:56.214466 12012 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:56.421307 12104 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20250627 13:56:56.564846 12243 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replayed 1/1 log segments. Stats: ops{read=1953 overwritten=0 applied=1953 ignored=0} inserts{seen=97550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:56:56.565703 12243 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap complete.
I20250627 13:56:56.567690 12243 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent bootstrapping tablet: real 8.649s user 7.683s sys 0.526s
I20250627 13:56:56.572850 12243 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:56.573976 12243 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e364b586e84dba94e1678b85d63274, State: Initialized, Role: FOLLOWER
I20250627 13:56:56.575120 12243 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1953, Last appended: 3.1953, Last appended by leader: 1953, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:56:56.589205 12243 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20250627 13:56:56.620172 12305 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 4 FOLLOWER]: Advancing to term 5
I20250627 13:56:56.626358 12305 raft_consensus.cc:1273] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 5 FOLLOWER]: Refusing update from remote peer 2cbadb2aefb24b38b5dca68b10368ddf: Log matching property violated. Preceding OpId in replica: term: 3 index: 1953. Preceding OpId from leader: term: 3 index: 1955. (index mismatch)
I20250627 13:56:56.629966 12392 consensus_queue.cc:1037] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1956, Last known committed idx: 1953, Time since last communication: 0.000s
I20250627 13:56:57.093266 12455 mvcc.cc:204] Tried to move back new op lower bound from 7172229597716992000 to 7172229590061539328. Current Snapshot: MvccSnapshot[applied={T|T < 7172229597651996672}]
I20250627 13:57:02.434556 12286 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:02.509822 12149 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:02.531742 12012 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:02.539968 11876 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:03.758985 10685 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.8.167.196:43821, user_credentials={real_user=slave}} blocked reactor thread for 101957us
I20250627 13:57:04.064254 10092 ts_manager.cc:284] Unset tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 from MAINTENANCE_MODE
I20250627 13:57:04.090216 10092 ts_manager.cc:284] Unset tserver state for 7397f03f0d404d6692a8318392f3ba7c from MAINTENANCE_MODE
I20250627 13:57:04.446934 10092 ts_manager.cc:284] Unset tserver state for 26e364b586e84dba94e1678b85d63274 from MAINTENANCE_MODE
I20250627 13:57:04.505936 10092 ts_manager.cc:284] Unset tserver state for 2cbadb2aefb24b38b5dca68b10368ddf from MAINTENANCE_MODE
I20250627 13:57:04.549328 11941 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:04.803920 12214 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:04.812273 12351 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:04.923640 12077 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
W20250627 13:57:06.788343 10712 meta_cache.cc:1261] Time spent looking up entry by key: real 0.077s user 0.001s sys 0.000s
I20250627 13:57:07.501812 10092 ts_manager.cc:295] Set tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 to MAINTENANCE_MODE
I20250627 13:57:07.508123 10086 ts_manager.cc:295] Set tserver state for 7397f03f0d404d6692a8318392f3ba7c to MAINTENANCE_MODE
I20250627 13:57:07.566421 10086 ts_manager.cc:295] Set tserver state for 2cbadb2aefb24b38b5dca68b10368ddf to MAINTENANCE_MODE
W20250627 13:57:09.439610 10713 meta_cache.cc:1261] Time spent looking up entry by key: real 0.066s user 0.004s sys 0.000s
I20250627 13:57:09.888466 10086 ts_manager.cc:295] Set tserver state for 26e364b586e84dba94e1678b85d63274 to MAINTENANCE_MODE
I20250627 13:57:10.056193 12012 tablet_service.cc:1423] Tablet server 7397f03f0d404d6692a8318392f3ba7c set to quiescing
I20250627 13:57:10.056484 12012 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:10.854753 11876 tablet_service.cc:1423] Tablet server 9e690d05ade74ce38ef7c57ca1ff4709 set to quiescing
I20250627 13:57:10.855068 11876 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:11.216270 12149 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:57:11.216533 12149 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:11.241015 12467 raft_consensus.cc:991] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: : Instructing follower 26e364b586e84dba94e1678b85d63274 to start an election
I20250627 13:57:11.241209 12467 raft_consensus.cc:1079] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 LEADER]: Signalling peer 26e364b586e84dba94e1678b85d63274 to start an election
I20250627 13:57:11.242563 12305 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "648e63fd67a54d22afac864a578a6912"
dest_uuid: "26e364b586e84dba94e1678b85d63274"
from {username='slave'} at 127.8.167.196:51725
I20250627 13:57:11.242997 12305 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:57:11.243249 12305 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 5 FOLLOWER]: Advancing to term 6
I20250627 13:57:11.246342 12305 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:11.248541 12305 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 6 election: Requested vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:57:11.253338 12305 raft_consensus.cc:1238] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Rejecting Update request from peer 2cbadb2aefb24b38b5dca68b10368ddf for earlier term 5. Current term is 6. Ops: [5.3424-5.3425]
I20250627 13:57:11.254168 12467 consensus_queue.cc:1046] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 }, Status: INVALID_TERM, Last received: 5.3423, Next index: 3424, Last known committed idx: 3423, Time since last communication: 0.000s
I20250627 13:57:11.254601 12566 raft_consensus.cc:3053] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 LEADER]: Stepping down as leader of term 5
I20250627 13:57:11.254698 12566 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:57:11.255016 12566 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3423, Committed index: 3423, Last appended: 5.3425, Last appended by leader: 3425, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:11.270906 12566 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 5 FOLLOWER]: Advancing to term 6
W20250627 13:57:11.275483 12411 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.024s ago.
W20250627 13:57:11.281893 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:57:11.290462 12031 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 6 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: true dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
I20250627 13:57:11.290709 12031 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 5 FOLLOWER]: Advancing to term 6
W20250627 13:57:11.291644 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:11.293799 12031 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 6 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:11.295338 12169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 6 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: true dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
I20250627 13:57:11.295765 12169 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 6 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:11.296350 12238 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e364b586e84dba94e1678b85d63274; no voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c
W20250627 13:57:11.301498 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:11.308012 12691 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250627 13:57:11.309044 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.316506 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.329998 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.339126 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.353699 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.368471 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.381510 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.399281 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.417342 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.435895 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.453703 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.479053 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.499421 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.520054 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.544073 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.569507 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.576877 12693 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:57:11.585256 12566 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:57:11.597157 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.625685 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:11.637126 12691 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:11.637427 12691 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:11.638835 12691 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:57:11.640180 12031 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 7 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:57:11.640597 12031 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 7 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:11.641067 12169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 7 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
I20250627 13:57:11.641361 12169 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 7 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:11.641852 12238 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e364b586e84dba94e1678b85d63274; no voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c
I20250627 13:57:11.642166 12691 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250627 13:57:11.653395 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.680332 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.711807 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.741084 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.771638 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.805457 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.840343 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.872074 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:11.911465 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:11.947440 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:11.985903 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:12.027694 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.067749 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.125031 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:12.131426 12691 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:12.131784 12691 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:12.132822 12691 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:57:12.133850 12169 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 7 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
I20250627 13:57:12.133848 12031 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "26e364b586e84dba94e1678b85d63274" candidate_term: 7 candidate_status { last_received { term: 5 index: 3423 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:57:12.134203 12169 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 7 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:12.134323 12031 raft_consensus.cc:2408] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26e364b586e84dba94e1678b85d63274 for term 7 because replica has last-logged OpId of term: 5 index: 3425, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3423.
I20250627 13:57:12.135088 12238 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e364b586e84dba94e1678b85d63274; no voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c
I20250627 13:57:12.135887 12691 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250627 13:57:12.172044 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.212954 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.256448 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:12.305276 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.349061 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.396173 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:12.447314 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.496933 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.543490 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:12.584904 12286 tablet_service.cc:1423] Tablet server 26e364b586e84dba94e1678b85d63274 set to quiescing
I20250627 13:57:12.585139 12286 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:12.593657 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.644958 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.697310 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:12.750021 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.801285 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:12.855696 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:12.906855 12706 raft_consensus.cc:668] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: failed to trigger leader election: Illegal state: leader elections are disabled
W20250627 13:57:12.913599 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:12.968904 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:13.025257 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:13.084219 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:13.141666 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:13.201947 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:13.266064 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:13.329254 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:13.392628 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:13.459533 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:13.525300 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:13.590167 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:13.661250 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:13.728991 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:13.810832 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:13.884995 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:13.901732 12149 tablet_service.cc:1423] Tablet server 2cbadb2aefb24b38b5dca68b10368ddf set to quiescing
I20250627 13:57:13.901985 12149 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:13.954210 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:57:13.984819 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 11814
I20250627 13:57:13.992115 11936 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:57:14.024531 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:14.082173 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 11814
W20250627 13:57:14.099596 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:14.103493 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.194:37449
--local_ip_for_outbound_sockets=127.8.167.194
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=37159
--webserver_interface=127.8.167.194
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:57:14.172847 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:14.247352 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:14.324810 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:14.402081 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:14.456089 12722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:14.456447 12722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:14.456602 12722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:14.464704 12722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:14.464987 12722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.194
I20250627 13:57:14.474952 12722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.194:37449
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.167.194
--webserver_port=37159
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.194
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:57:14.476742 12722 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20250627 13:57:14.478677 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:14.479310 12722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:14.491688 12728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:14.491765 12730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:14.491724 12727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:14.492656 12722 server_base.cc:1048] running on GCE node
I20250627 13:57:14.493351 12722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:14.494400 12722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:14.495641 12722 hybrid_clock.cc:648] HybridClock initialized: now 1751032634495598 us; error 38 us; skew 500 ppm
I20250627 13:57:14.495971 12722 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:14.498391 12722 webserver.cc:469] Webserver started at http://127.8.167.194:37159/ using document root <none> and password file <none>
I20250627 13:57:14.499312 12722 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:14.499439 12722 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:14.505224 12722 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20250627 13:57:14.508128 12736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:14.509545 12722 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250627 13:57:14.509723 12722 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
uuid: "9e690d05ade74ce38ef7c57ca1ff4709"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:57:14.510533 12722 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250627 13:57:14.559175 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:14.559906 12722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:14.560788 12722 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:14.561033 12722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:14.561992 12722 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:14.563771 12722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:57:14.563851 12722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:14.563943 12722 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:57:14.563987 12722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:14.598244 12722 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.194:37449
I20250627 13:57:14.598304 12848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.194:37449 every 8 connection(s)
I20250627 13:57:14.599821 12722 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-1/data/info.pb
I20250627 13:57:14.609728 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 12722
I20250627 13:57:14.609983 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 11948
I20250627 13:57:14.613566 12849 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:57:14.614020 12849 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:14.615001 12849 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:14.616655 10092 ts_manager.cc:194] Re-registered known tserver with Master: 9e690d05ade74ce38ef7c57ca1ff4709 (127.8.167.194:37449)
I20250627 13:57:14.617522 12072 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:14.617600 10092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.194:54899
W20250627 13:57:14.912833 11992 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:44824: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:57:14.913579 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 11948
I20250627 13:57:14.963267 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.193:37697
--local_ip_for_outbound_sockets=127.8.167.193
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=41461
--webserver_interface=127.8.167.193
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:57:14.993916 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.076681 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:15.161852 10685 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.193:37697: connect: Connection refused (error 111)
W20250627 13:57:15.246877 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.309991 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:15.310374 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:15.310519 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:15.318795 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:15.319031 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.193
I20250627 13:57:15.328541 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.193:37697
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.167.193
--webserver_port=41461
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.193
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:57:15.330193 12856 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:15.331964 12856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:15.336107 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
W20250627 13:57:15.344650 12862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:15.344712 12863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:15.344687 12865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:15.345595 12856 server_base.cc:1048] running on GCE node
I20250627 13:57:15.346298 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:15.347306 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:15.348567 12856 hybrid_clock.cc:648] HybridClock initialized: now 1751032635348489 us; error 51 us; skew 500 ppm
I20250627 13:57:15.348886 12856 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:15.351406 12856 webserver.cc:469] Webserver started at http://127.8.167.193:41461/ using document root <none> and password file <none>
I20250627 13:57:15.352253 12856 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:15.352368 12856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:15.357960 12856 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20250627 13:57:15.361040 12871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:15.362660 12856 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250627 13:57:15.362862 12856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
uuid: "7397f03f0d404d6692a8318392f3ba7c"
format_stamp: "Formatted at 2025-06-27 13:56:03 on dist-test-slave-1xs4"
I20250627 13:57:15.363579 12856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:15.383807 12856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:15.384761 12856 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:15.385047 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:15.386147 12856 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:15.388428 12878 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:57:15.394507 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:57:15.394657 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20250627 13:57:15.394809 12856 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:57:15.398141 12856 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:57:15.398277 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20250627 13:57:15.398782 12878 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap starting.
I20250627 13:57:15.447935 12856 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.193:37697
I20250627 13:57:15.448029 12984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.193:37697 every 8 connection(s)
I20250627 13:57:15.450127 12856 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-0/data/info.pb
I20250627 13:57:15.458812 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 12856
I20250627 13:57:15.459022 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 12085
I20250627 13:57:15.466794 12985 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:57:15.467211 12985 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:15.468099 12208 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:15.468372 12985 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:15.471808 10092 ts_manager.cc:194] Re-registered known tserver with Master: 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697)
I20250627 13:57:15.473817 10092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.193:41919
W20250627 13:57:15.506371 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:15.620692 12849 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:57:15.780792 12127 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:36890: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:15.782209 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 12085
I20250627 13:57:15.857977 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.196:43821
--local_ip_for_outbound_sockets=127.8.167.196
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=34547
--webserver_interface=127.8.167.196
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:57:15.893075 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.894289 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.913703 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.917912 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.948156 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.948168 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.978935 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.979862 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:15.992424 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.041958 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.043922 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.105669 12266 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.107007 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.200966 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.204031 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.230500 10685 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.196:43821: connect: Connection refused (error 111)
W20250627 13:57:16.252478 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.255590 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.258828 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.262653 12992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:16.262988 12992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:16.263139 12992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:16.271188 12992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:16.271406 12992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.196
I20250627 13:57:16.281323 12992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.196:43821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.8.167.196
--webserver_port=34547
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.196
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:57:16.283013 12992 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:16.284965 12992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:16.299360 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:16.299780 12992 server_base.cc:1048] running on GCE node
W20250627 13:57:16.299443 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:16.299984 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:16.300877 12992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:16.302131 12992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:16.303391 12992 hybrid_clock.cc:648] HybridClock initialized: now 1751032636303342 us; error 36 us; skew 500 ppm
I20250627 13:57:16.303786 12992 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:16.306763 12992 webserver.cc:469] Webserver started at http://127.8.167.196:34547/ using document root <none> and password file <none>
I20250627 13:57:16.307801 12992 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:16.307927 12992 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:16.315202 12992 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250627 13:57:16.318970 13007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:16.320523 12992 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:57:16.320709 12992 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
uuid: "2cbadb2aefb24b38b5dca68b10368ddf"
format_stamp: "Formatted at 2025-06-27 13:56:05 on dist-test-slave-1xs4"
I20250627 13:57:16.321491 12992 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:16.344950 12992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:16.345968 12992 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:16.346256 12992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:16.347229 12992 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:16.349906 13014 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:57:16.358163 12992 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:57:16.358283 12992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20250627 13:57:16.358438 12992 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:57:16.361662 12992 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:57:16.361812 12992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250627 13:57:16.362422 13014 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap starting.
W20250627 13:57:16.375257 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.382714 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:16.437888 12992 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.196:43821
I20250627 13:57:16.437935 13120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.196:43821 every 8 connection(s)
I20250627 13:57:16.439749 12992 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-3/data/info.pb
I20250627 13:57:16.444185 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 12992
I20250627 13:57:16.444423 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 12222
I20250627 13:57:16.476224 12878 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:16.477361 12985 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
W20250627 13:57:16.515935 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.517527 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
W20250627 13:57:16.518790 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:16.546212 12345 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:16.578678 13121 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:57:16.579227 13121 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:16.580657 13121 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:16.585117 10092 ts_manager.cc:194] Re-registered known tserver with Master: 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821)
I20250627 13:57:16.587366 10092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.196:39735
W20250627 13:57:16.905380 12265 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:54828: Illegal state: replica 26e364b586e84dba94e1678b85d63274 is not leader of this config: current role FOLLOWER
I20250627 13:57:16.908119 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 12222
W20250627 13:57:16.958432 10687 connection.cc:537] client connection to 127.8.167.195:34991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:16.959208 8863 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
/tmp/dist-test-taskHxmGc1/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.167.195:34991
--local_ip_for_outbound_sockets=127.8.167.195
--tserver_master_addrs=127.8.167.254:38489
--webserver_port=35703
--webserver_interface=127.8.167.195
--builtin_ntp_servers=127.8.167.212:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:57:17.254426 10687 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
W20250627 13:57:17.335814 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:17.336174 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:57:17.336329 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:17.344985 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:57:17.345219 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.167.195
I20250627 13:57:17.357336 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.167.212:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.167.195:34991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.167.195
--webserver_port=35703
--tserver_master_addrs=127.8.167.254:38489
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.167.195
--log_dir=/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:57:17.359115 13130 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:57:17.361065 13130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:57:17.377848 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:17.377887 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:57:17.379704 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:57:17.381760 13130 server_base.cc:1048] running on GCE node
I20250627 13:57:17.382578 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:57:17.384053 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:57:17.385330 13130 hybrid_clock.cc:648] HybridClock initialized: now 1751032637385267 us; error 49 us; skew 500 ppm
I20250627 13:57:17.385738 13130 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:57:17.388942 13130 webserver.cc:469] Webserver started at http://127.8.167.195:35703/ using document root <none> and password file <none>
I20250627 13:57:17.389981 13130 fs_manager.cc:362] Metadata directory not provided
I20250627 13:57:17.390151 13130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:57:17.399649 13130 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.004s
I20250627 13:57:17.403544 13144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:57:17.405550 13130 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:57:17.405746 13130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data,/tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
uuid: "26e364b586e84dba94e1678b85d63274"
format_stamp: "Formatted at 2025-06-27 13:56:04 on dist-test-slave-1xs4"
I20250627 13:57:17.406800 13130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:57:17.430845 13130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:57:17.431959 13130 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:57:17.432266 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:57:17.433611 13130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:57:17.436410 13151 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:57:17.445761 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:57:17.445899 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20250627 13:57:17.446059 13130 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:57:17.451531 13151 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap starting.
I20250627 13:57:17.458123 13130 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:57:17.458254 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.002s sys 0.003s
I20250627 13:57:17.593340 13121 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:57:17.947638 13014 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Log is configured to *not* fsync() on all Append() calls
W20250627 13:57:18.345114 10687 meta_cache.cc:302] tablet 648e63fd67a54d22afac864a578a6912: replica 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991) has failed: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111)
I20250627 13:57:18.367890 13258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.167.195:34991 every 8 connection(s)
I20250627 13:57:18.369220 13130 rpc_server.cc:307] RPC server started. Bound to: 127.8.167.195:34991
I20250627 13:57:18.371471 13130 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0/minicluster-data/ts-2/data/info.pb
I20250627 13:57:18.378789 8863 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu as pid 13130
I20250627 13:57:18.444617 13259 heartbeater.cc:344] Connected to a master server at 127.8.167.254:38489
I20250627 13:57:18.445111 13259 heartbeater.cc:461] Registering TS with master...
I20250627 13:57:18.446400 13259 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:18.454952 10092 ts_manager.cc:194] Re-registered known tserver with Master: 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:57:18.456470 10092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.167.195:43469
I20250627 13:57:19.112469 13151 log.cc:826] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Log is configured to *not* fsync() on all Append() calls
I20250627 13:57:19.462067 13259 heartbeater.cc:499] Master 127.8.167.254:38489 was elected leader, sending a full tablet report...
I20250627 13:57:19.537853 13056 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:19.563195 13193 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:19.598304 12784 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:19.599085 12920 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:21.487565 12878 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replaying log segment 1/1 (2.27M/5.88M this segment, stats: ops{read=1312 overwritten=0 applied=1312 ignored=0} inserts{seen=65500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:22.958381 13014 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replaying log segment 1/1 (2.62M/5.93M this segment, stats: ops{read=1513 overwritten=0 applied=1513 ignored=0} inserts{seen=75550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:24.121999 13151 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replaying log segment 1/1 (2.20M/5.80M this segment, stats: ops{read=1271 overwritten=0 applied=1271 ignored=0} inserts{seen=63450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:25.940016 13193 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:26.019287 13056 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:26.029356 12784 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:26.039034 12920 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:26.488262 12878 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replaying log segment 1/1 (4.40M/5.88M this segment, stats: ops{read=2554 overwritten=0 applied=2553 ignored=0} inserts{seen=127500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:27.958662 13014 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replaying log segment 1/1 (5.15M/5.93M this segment, stats: ops{read=2977 overwritten=0 applied=2976 ignored=0} inserts{seen=148650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:29.123512 13151 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replaying log segment 1/1 (4.72M/5.80M this segment, stats: ops{read=2780 overwritten=0 applied=2779 ignored=0} inserts{seen=138800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:29.410498 12878 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3425 overwritten=0 applied=3423 ignored=0} inserts{seen=171000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250627 13:57:29.411151 12878 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Bootstrap complete.
I20250627 13:57:29.412871 12878 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent bootstrapping tablet: real 14.015s user 13.094s sys 0.779s
I20250627 13:57:29.417554 12878 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:29.420486 12878 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7397f03f0d404d6692a8318392f3ba7c, State: Initialized, Role: FOLLOWER
I20250627 13:57:29.421409 12878 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3423, Last appended: 5.3425, Last appended by leader: 3425, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:29.423858 12878 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20250627 13:57:29.660291 13336 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:29.660672 13336 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:29.671268 13336 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:57:29.678534 13014 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap replayed 1/1 log segments. Stats: ops{read=3425 overwritten=0 applied=3423 ignored=0} inserts{seen=171000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250627 13:57:29.679459 13014 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Bootstrap complete.
I20250627 13:57:29.681908 13014 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent bootstrapping tablet: real 13.320s user 12.272s sys 0.644s
I20250627 13:57:29.684562 13076 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" is_pre_election: true
I20250627 13:57:29.687868 13214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "7397f03f0d404d6692a8318392f3ba7c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
W20250627 13:57:29.687386 12873 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 2cbadb2aefb24b38b5dca68b10368ddf (127.8.167.196:43821): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:57:29.690838 13014 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
W20250627 13:57:29.691815 12875 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:57:29.692250 12875 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274, 2cbadb2aefb24b38b5dca68b10368ddf
I20250627 13:57:29.693017 13336 raft_consensus.cc:2747] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250627 13:57:29.694968 13014 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Initialized, Role: FOLLOWER
I20250627 13:57:29.695849 13014 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3423, Last appended: 5.3425, Last appended by leader: 3425, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
W20250627 13:57:29.698863 12899 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41858: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
I20250627 13:57:29.703998 13014 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf: Time spent starting tablet: real 0.022s user 0.014s sys 0.004s
I20250627 13:57:30.061573 13342 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:57:30.061892 13342 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:30.063974 13342 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:57:30.072034 13214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274" is_pre_election: true
I20250627 13:57:30.072417 12940 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c" is_pre_election: true
I20250627 13:57:30.072788 12940 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 6.
W20250627 13:57:30.072716 13011 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 pre-election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
I20250627 13:57:30.073254 13009 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274
I20250627 13:57:30.073621 13342 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250627 13:57:30.073756 13342 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:57:30.073848 13342 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 6 FOLLOWER]: Advancing to term 7
I20250627 13:57:30.078716 13342 raft_consensus.cc:513] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:30.079448 13342 leader_election.cc:290] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 election: Requested vote from peers 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697), 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991)
I20250627 13:57:30.079917 12940 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "7397f03f0d404d6692a8318392f3ba7c"
I20250627 13:57:30.080124 12940 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 6 FOLLOWER]: Advancing to term 7
I20250627 13:57:30.080350 13214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "648e63fd67a54d22afac864a578a6912" candidate_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3425 } } ignore_live_leader: false dest_uuid: "26e364b586e84dba94e1678b85d63274"
W20250627 13:57:30.080929 13011 leader_election.cc:343] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 election: Tablet error from VoteRequest() call to peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Illegal state: must be running to vote when last-logged opid is not known
W20250627 13:57:30.081771 13035 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51472: Illegal state: replica 2cbadb2aefb24b38b5dca68b10368ddf is not leader of this config: current role FOLLOWER
I20250627 13:57:30.084918 12940 raft_consensus.cc:2466] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbadb2aefb24b38b5dca68b10368ddf in term 7.
I20250627 13:57:30.085407 13009 leader_election.cc:304] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cbadb2aefb24b38b5dca68b10368ddf, 7397f03f0d404d6692a8318392f3ba7c; no voters: 26e364b586e84dba94e1678b85d63274
I20250627 13:57:30.085748 13342 raft_consensus.cc:2802] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 7 FOLLOWER]: Leader election won for term 7
I20250627 13:57:30.086262 13342 raft_consensus.cc:695] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [term 7 LEADER]: Becoming Leader. State: Replica: 2cbadb2aefb24b38b5dca68b10368ddf, State: Running, Role: LEADER
I20250627 13:57:30.087002 13342 consensus_queue.cc:237] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3423, Committed index: 3423, Last appended: 5.3425, Last appended by leader: 3425, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:30.091789 10092 catalog_manager.cc:5582] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } health_report { overall_health: UNKNOWN } } }
W20250627 13:57:30.173377 13011 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:30.206270 12940 raft_consensus.cc:1273] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 7 FOLLOWER]: Refusing update from remote peer 2cbadb2aefb24b38b5dca68b10368ddf: Log matching property violated. Preceding OpId in replica: term: 5 index: 3425. Preceding OpId from leader: term: 7 index: 3426. (index mismatch)
I20250627 13:57:30.207218 13342 consensus_queue.cc:1035] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3426, Last known committed idx: 3423, Time since last communication: 0.000s
W20250627 13:57:30.270531 12899 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41858: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:30.271054 12900 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41858: Illegal state: replica 7397f03f0d404d6692a8318392f3ba7c is not leader of this config: current role FOLLOWER
W20250627 13:57:30.680966 13011 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20250627 13:57:31.197933 13011 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20250627 13:57:31.215672 13151 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap replayed 1/1 log segments. Stats: ops{read=3423 overwritten=0 applied=3423 ignored=0} inserts{seen=171000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:57:31.216616 13151 tablet_bootstrap.cc:492] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Bootstrap complete.
I20250627 13:57:31.218663 13151 ts_tablet_manager.cc:1397] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent bootstrapping tablet: real 13.768s user 12.773s sys 0.774s
I20250627 13:57:31.222666 13151 raft_consensus.cc:357] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:31.223273 13151 raft_consensus.cc:738] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e364b586e84dba94e1678b85d63274, State: Initialized, Role: FOLLOWER
I20250627 13:57:31.224140 13151 consensus_queue.cc:260] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3423, Last appended: 5.3423, Last appended by leader: 3423, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbadb2aefb24b38b5dca68b10368ddf" member_type: VOTER last_known_addr { host: "127.8.167.196" port: 43821 } } peers { permanent_uuid: "7397f03f0d404d6692a8318392f3ba7c" member_type: VOTER last_known_addr { host: "127.8.167.193" port: 37697 } } peers { permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 } }
I20250627 13:57:31.228507 13151 ts_tablet_manager.cc:1428] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250627 13:57:31.321787 13214 raft_consensus.cc:3058] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 6 FOLLOWER]: Advancing to term 7
I20250627 13:57:31.329216 13214 raft_consensus.cc:1273] T 648e63fd67a54d22afac864a578a6912 P 26e364b586e84dba94e1678b85d63274 [term 7 FOLLOWER]: Refusing update from remote peer 2cbadb2aefb24b38b5dca68b10368ddf: Log matching property violated. Preceding OpId in replica: term: 5 index: 3423. Preceding OpId from leader: term: 5 index: 3425. (index mismatch)
I20250627 13:57:31.335837 13360 consensus_queue.cc:1037] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "26e364b586e84dba94e1678b85d63274" member_type: VOTER last_known_addr { host: "127.8.167.195" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3426, Last known committed idx: 3423, Time since last communication: 0.000s
I20250627 13:57:32.080945 13366 mvcc.cc:204] Tried to move back new op lower bound from 7172229739848314880 to 7172229734760071168. Current Snapshot: MvccSnapshot[applied={T|T < 7172229739848314880 or (T in {7172229739848314880})}]
I20250627 13:57:32.630440 13056 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:32.632920 12920 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.667363 13193 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.702142 12784 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:33.827771 10092 ts_manager.cc:284] Unset tserver state for 9e690d05ade74ce38ef7c57ca1ff4709 from MAINTENANCE_MODE
I20250627 13:57:33.885327 10095 ts_manager.cc:284] Unset tserver state for 7397f03f0d404d6692a8318392f3ba7c from MAINTENANCE_MODE
I20250627 13:57:34.143638 10095 ts_manager.cc:284] Unset tserver state for 26e364b586e84dba94e1678b85d63274 from MAINTENANCE_MODE
I20250627 13:57:34.249179 12985 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:34.338474 10095 ts_manager.cc:284] Unset tserver state for 2cbadb2aefb24b38b5dca68b10368ddf from MAINTENANCE_MODE
I20250627 13:57:34.582937 13259 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:34.624603 13121 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
I20250627 13:57:34.672199 12849 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
W20250627 13:57:34.917910 13009 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.167.193:37697, user_credentials={real_user=slave}} blocked reactor thread for 50109.2us
W20250627 13:57:35.098462 10712 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.001s sys 0.003s
I20250627 13:57:35.254979 12985 heartbeater.cc:507] Master 127.8.167.254:38489 requested a full tablet report, sending...
W20250627 13:57:36.368252 10685 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.8.167.196:43821, user_credentials={real_user=slave}} blocked reactor thread for 67131.1us
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:750: Failure
Value of: s.ok()
Actual: true
Expected: false
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
I20250627 13:57:36.586338 8863 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250627 13:57:36.586582 8863 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 7397f03f0d404d6692a8318392f3ba7c and pid 12856
************************ BEGIN STACKS **************************
[New LWP 12858]
[New LWP 12859]
[New LWP 12860]
[New LWP 12861]
[New LWP 12867]
[New LWP 12868]
[New LWP 12869]
[New LWP 12872]
[New LWP 12873]
[New LWP 12874]
[New LWP 12875]
[New LWP 12876]
[New LWP 12877]
[New LWP 12879]
[New LWP 12880]
[New LWP 12881]
[New LWP 12882]
[New LWP 12883]
[New LWP 12884]
[New LWP 12885]
[New LWP 12886]
[New LWP 12887]
[New LWP 12888]
[New LWP 12889]
[New LWP 12890]
[New LWP 12891]
[New LWP 12892]
[New LWP 12893]
[New LWP 12894]
[New LWP 12895]
[New LWP 12896]
[New LWP 12897]
[New LWP 12898]
[New LWP 12899]
[New LWP 12900]
[New LWP 12901]
[New LWP 12902]
[New LWP 12903]
[New LWP 12904]
[New LWP 12905]
[New LWP 12906]
[New LWP 12907]
[New LWP 12908]
[New LWP 12909]
[New LWP 12910]
[New LWP 12911]
[New LWP 12912]
[New LWP 12913]
[New LWP 12914]
[New LWP 12915]
[New LWP 12916]
[New LWP 12917]
[New LWP 12918]
[New LWP 12919]
[New LWP 12920]
[New LWP 12921]
[New LWP 12922]
[New LWP 12923]
[New LWP 12924]
[New LWP 12925]
[New LWP 12926]
[New LWP 12927]
[New LWP 12928]
[New LWP 12929]
[New LWP 12930]
[New LWP 12931]
[New LWP 12932]
[New LWP 12933]
[New LWP 12934]
[New LWP 12935]
[New LWP 12936]
[New LWP 12937]
[New LWP 12938]
[New LWP 12939]
[New LWP 12940]
[New LWP 12941]
[New LWP 12942]
[New LWP 12943]
[New LWP 12944]
[New LWP 12945]
[New LWP 12946]
[New LWP 12947]
[New LWP 12948]
[New LWP 12949]
[New LWP 12950]
[New LWP 12951]
[New LWP 12952]
[New LWP 12953]
[New LWP 12954]
[New LWP 12955]
[New LWP 12956]
[New LWP 12957]
[New LWP 12958]
[New LWP 12959]
[New LWP 12960]
[New LWP 12961]
[New LWP 12962]
[New LWP 12963]
[New LWP 12964]
[New LWP 12965]
[New LWP 12966]
[New LWP 12967]
[New LWP 12968]
[New LWP 12969]
[New LWP 12970]
[New LWP 12971]
[New LWP 12972]
[New LWP 12973]
[New LWP 12974]
[New LWP 12975]
[New LWP 12976]
[New LWP 12977]
[New LWP 12978]
[New LWP 12979]
[New LWP 12980]
[New LWP 12981]
[New LWP 12982]
[New LWP 12983]
[New LWP 12984]
[New LWP 12985]
[New LWP 12986]
[New LWP 13353]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007efeb0f5bd50 in ?? ()
Id Target Id Frame
* 1 LWP 12856 "kudu" 0x00007efeb0f5bd50 in ?? ()
2 LWP 12858 "kudu" 0x00007efeb0f57fb9 in ?? ()
3 LWP 12859 "kudu" 0x00007efeb0f57fb9 in ?? ()
4 LWP 12860 "kudu" 0x00007efeb0f57fb9 in ?? ()
5 LWP 12861 "kernel-watcher-" 0x00007efeb0f57fb9 in ?? ()
6 LWP 12867 "ntp client-1286" 0x00007efeb0f5b9e2 in ?? ()
7 LWP 12868 "file cache-evic" 0x00007efeb0f57fb9 in ?? ()
8 LWP 12869 "sq_acceptor" 0x00007efea9c7dcb9 in ?? ()
9 LWP 12872 "rpc reactor-128" 0x00007efea9c8aa47 in ?? ()
10 LWP 12873 "rpc reactor-128" 0x00007efea9c8aa47 in ?? ()
11 LWP 12874 "rpc reactor-128" 0x00007efea9c8aa47 in ?? ()
12 LWP 12875 "rpc reactor-128" 0x00007efea9c8aa47 in ?? ()
13 LWP 12876 "MaintenanceMgr " 0x00007efeb0f57ad3 in ?? ()
14 LWP 12877 "txn-status-mana" 0x00007efeb0f57fb9 in ?? ()
15 LWP 12879 "collect_and_rem" 0x00007efeb0f57fb9 in ?? ()
16 LWP 12880 "tc-session-exp-" 0x00007efeb0f57fb9 in ?? ()
17 LWP 12881 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
18 LWP 12882 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
19 LWP 12883 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
20 LWP 12884 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
21 LWP 12885 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
22 LWP 12886 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
23 LWP 12887 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
24 LWP 12888 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
25 LWP 12889 "rpc worker-1288" 0x00007efeb0f57ad3 in ?? ()
26 LWP 12890 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
27 LWP 12891 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
28 LWP 12892 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
29 LWP 12893 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
30 LWP 12894 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
31 LWP 12895 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
32 LWP 12896 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
33 LWP 12897 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
34 LWP 12898 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
35 LWP 12899 "rpc worker-1289" 0x00007efeb0f57ad3 in ?? ()
36 LWP 12900 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
37 LWP 12901 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
38 LWP 12902 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
39 LWP 12903 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
40 LWP 12904 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
41 LWP 12905 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
42 LWP 12906 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
43 LWP 12907 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
44 LWP 12908 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
45 LWP 12909 "rpc worker-1290" 0x00007efeb0f57ad3 in ?? ()
46 LWP 12910 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
47 LWP 12911 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
48 LWP 12912 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
49 LWP 12913 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
50 LWP 12914 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
51 LWP 12915 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
52 LWP 12916 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
53 LWP 12917 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
54 LWP 12918 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
55 LWP 12919 "rpc worker-1291" 0x00007efeb0f57ad3 in ?? ()
56 LWP 12920 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
57 LWP 12921 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
58 LWP 12922 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
59 LWP 12923 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
60 LWP 12924 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
61 LWP 12925 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
62 LWP 12926 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
63 LWP 12927 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
64 LWP 12928 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
65 LWP 12929 "rpc worker-1292" 0x00007efeb0f57ad3 in ?? ()
66 LWP 12930 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
67 LWP 12931 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
68 LWP 12932 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
69 LWP 12933 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
70 LWP 12934 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
71 LWP 12935 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
72 LWP 12936 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
73 LWP 12937 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
74 LWP 12938 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
75 LWP 12939 "rpc worker-1293" 0x00007efeb0f57ad3 in ?? ()
76 LWP 12940 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
77 LWP 12941 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
78 LWP 12942 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
79 LWP 12943 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
80 LWP 12944 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
81 LWP 12945 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
82 LWP 12946 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
83 LWP 12947 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
84 LWP 12948 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
85 LWP 12949 "rpc worker-1294" 0x00007efeb0f57ad3 in ?? ()
86 LWP 12950 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
87 LWP 12951 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
88 LWP 12952 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
89 LWP 12953 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
90 LWP 12954 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
91 LWP 12955 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
92 LWP 12956 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
93 LWP 12957 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
94 LWP 12958 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
95 LWP 12959 "rpc worker-1295" 0x00007efeb0f57ad3 in ?? ()
96 LWP 12960 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
97 LWP 12961 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
98 LWP 12962 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
99 LWP 12963 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
100 LWP 12964 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
101 LWP 12965 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
102 LWP 12966 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
103 LWP 12967 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
104 LWP 12968 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
105 LWP 12969 "rpc worker-1296" 0x00007efeb0f57ad3 in ?? ()
106 LWP 12970 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
107 LWP 12971 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
108 LWP 12972 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
109 LWP 12973 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
110 LWP 12974 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
111 LWP 12975 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
112 LWP 12976 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
113 LWP 12977 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
114 LWP 12978 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
115 LWP 12979 "rpc worker-1297" 0x00007efeb0f57ad3 in ?? ()
116 LWP 12980 "rpc worker-1298" 0x00007efeb0f57ad3 in ?? ()
117 LWP 12981 "diag-logger-129" 0x00007efeb0f57fb9 in ?? ()
118 LWP 12982 "result-tracker-" 0x00007efeb0f57fb9 in ?? ()
119 LWP 12983 "excess-log-dele" 0x00007efeb0f57fb9 in ?? ()
120 LWP 12984 "acceptor-12984" 0x00007efea9c8c0c7 in ?? ()
121 LWP 12985 "heartbeat-12985" 0x00007efeb0f57fb9 in ?? ()
122 LWP 12986 "maintenance_sch" 0x00007efeb0f57fb9 in ?? ()
123 LWP 13353 "wal-append [wor" 0x00007efeb0f57fb9 in ?? ()
Thread 123 (LWP 13353):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x0000602000862bf8 in ?? ()
#2 0x00000000000003bf in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061a0005575b4 in ?? ()
#5 0x00007efe629c7110 in ?? ()
#6 0x000000000000077f in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007efe629c7130 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 12986):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efe6578f6e0 in ?? ()
#2 0x0000000000000057 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007efe6578f730 in ?? ()
#6 0x00000000000000ae in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12985):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0xb1139450f4d65aa7 in ?? ()
#2 0x0000000000000017 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000019e84 in ?? ()
#5 0x00007efe65fa75f0 in ?? ()
#6 0x000000000000002f in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007efe65fa7610 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007efe65fa7690 in ?? ()
#12 0x00000fdfccbf4ed4 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 12984):
#0 0x00007efea9c8c0c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007efe667ced90 in ?? ()
#3 0x00007efe667cedc0 in ?? ()
#4 0x00007efe667ceec0 in ?? ()
#5 0x00007efe667cedb0 in ?? ()
#6 0x00007efe667cee20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007efe667cef40 in ?? ()
#11 0x00007efe667ce67c in ?? ()
#12 0x00000032667ce5f0 in ?? ()
#13 0x00007efe65fd2000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 12983):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efe66fe7f40 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 12982):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efe67801100 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007efe678010f0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12981):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12980):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000007 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b684c in ?? ()
#4 0x00007efe68830e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe68830eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b6800 in ?? ()
#9 0x00007efeb0f57770 in ?? ()
#10 0x00007efe68830eb0 in ?? ()
#11 0x00000fdfcd1061d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 12979):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000005 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007efe69048e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe69048eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007efeb0f57770 in ?? ()
#10 0x00007efe69048eb0 in ?? ()
#11 0x00000fdfcd2091d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 12978):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12977):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12976):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12975):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12974):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12973):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12972):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12971):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12970):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12969):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12968):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12967):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12966):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12965):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12964):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12963):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12962):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12961):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12960):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12959):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12958):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12957):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12956):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12955):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12954):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12953):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12952):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12951):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12950):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12949):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12948):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12947):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12946):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12945):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12944):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12943):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12942):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12941):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12940):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x00000000000001aa in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007efe7cc01e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe7cc01eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 12939):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000166 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007efe7d419e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe7d419eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 12938):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 12937):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12936):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12935):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12934):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12933):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12932):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12931):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12930):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12929):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 12928):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 12927):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 12926):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 12925):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 12924):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 12923):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 12922):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 12921):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 12920):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007efe86de2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe86de2eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007efeb0f57770 in ?? ()
#10 0x00007efe86de2eb0 in ?? ()
#11 0x00000fdfd0dbc5d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 12919):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 12918):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 12917):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 12916):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 12915):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 12914):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 12913):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 12912):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 12911):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 12910):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 12909):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 12908):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 12907):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 12906):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 12905):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 12904):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 12903):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 12902):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 12901):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 12900):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x000000000000002a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007efe90fc3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe90fc3eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 12899):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000065 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007efe917dbe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe917dbeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007efeb0f57770 in ?? ()
#10 0x00007efe917dbeb0 in ?? ()
#11 0x00000fdfd22fb7d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 12898):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007efe91ff3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efe91ff3eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007efeb0f57770 in ?? ()
#10 0x00007efe91ff3eb0 in ?? ()
#11 0x00000fdfd23fe7d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 12897):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 12896):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 12895):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 12894):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 12893):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 12892):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 12891):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 12890):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 12889):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 12888):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 12887):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 12886):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 12885):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 12884):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 12883):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 12882):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 12881):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 12880):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efe9b1d6cc0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007efe9b1d6cb0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 12879):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 12877):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efe9ca5b240 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007efe9ca5b230 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 12876):
#0 0x00007efeb0f57ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 12875):
#0 0x00007efea9c8aa47 in ?? ()
#1 0x00007efe9daaa320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007efe9daaa310 in ?? ()
#4 0x00007efe9daaa520 in ?? ()
#5 0x00007efe9daaa360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007efe9daaa3e0 in ?? ()
#8 0x00007efeac2fd25d in ?? ()
#9 0x3fb95b795a5bc000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000064800000 in ?? ()
#14 0x00007efe00000000 in ?? ()
#15 0x41da17a845ddd369 in ?? ()
#16 0x00000fe053b4d47c in ?? ()
#17 0x00007efe9daaa3c0 in ?? ()
#18 0x00007efeac301ba3 in ?? ()
#19 0x00007efe9daaa390 in ?? ()
#20 0x3fb95b795a5bc000 in ?? ()
#21 0x000000009daaa3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb95b795a5bc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 12874):
#0 0x00007efea9c8aa47 in ?? ()
#1 0x00007efe9e2b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007efe9e2b3310 in ?? ()
#4 0x00007efe9e2b3520 in ?? ()
#5 0x00007efe9e2b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007efe9e2b33e0 in ?? ()
#8 0x00007efeac2fd25d in ?? ()
#9 0x3fb0d3b35b61a000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x000000005a000000 in ?? ()
#14 0x00007efe00000000 in ?? ()
#15 0x41da17a845ddd369 in ?? ()
#16 0x00000fe053c4e67c in ?? ()
#17 0x00007efe9e2b33c0 in ?? ()
#18 0x00007efeac301ba3 in ?? ()
#19 0x00007efe9e2b3390 in ?? ()
#20 0x3fb0d3b35b61a000 in ?? ()
#21 0x000000009e2b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb0d3b35b61a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 12873):
#0 0x00007efea9c8aa47 in ?? ()
#1 0x00007efe9eab4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007efe9eab4310 in ?? ()
#4 0x00007efe9eab4520 in ?? ()
#5 0x00007efe9eab4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007efe9eab43e0 in ?? ()
#8 0x00007efeac2fd25d in ?? ()
#9 0x3fb957a28f782000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000024000000 in ?? ()
#14 0x00007efe00000000 in ?? ()
#15 0x41da17a845ddd368 in ?? ()
#16 0x00000fe053d4e87c in ?? ()
#17 0x00007efe9eab43c0 in ?? ()
#18 0x00007efeac301ba3 in ?? ()
#19 0x00007efe9eab4390 in ?? ()
#20 0x3fb957a28f782000 in ?? ()
#21 0x000000009eab43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb957a28f782000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 12872):
#0 0x00007efea9c8aa47 in ?? ()
#1 0x00007efea0aad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007efea0aad310 in ?? ()
#4 0x00007efea0aad520 in ?? ()
#5 0x00007efea0aad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007efea0aad3e0 in ?? ()
#8 0x00007efeac2fd25d in ?? ()
#9 0x3fb983beb3568000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000007000000 in ?? ()
#14 0x00007efe00000000 in ?? ()
#15 0x41da17a845ddd36c in ?? ()
#16 0x00000fe05414da7c in ?? ()
#17 0x00007efea0aad3c0 in ?? ()
#18 0x00007efeac301ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 12869):
#0 0x00007efea9c7dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007efea24b77c8 in ?? ()
#3 0x000060200001c9d0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 12868):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efea1cb60c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 12867):
#0 0x00007efeb0f5b9e2 in ?? ()
#1 0x00007efea14b3ba0 in ?? ()
#2 0x00007efea14b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007efea14b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007efea14b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007efea14b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007efeb686eedc in ?? ()
#12 0x00007efeb685e8d9 in ?? ()
#13 0x00007efeb6862ec6 in ?? ()
#14 0x00007efeb6867900 in ?? ()
#15 0x00007efeb6867729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007efeadb334de in ?? ()
#18 0x00007efeb0f516db in ?? ()
#19 0x00000fe05428e684 in ?? ()
#20 0x00007efea14b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12861):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x00007efea34b8c80 in ?? ()
#2 0x000000000000006d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007efea34b8c70 in ?? ()
#6 0x00000000000000da in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12860):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12859):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12858):
#0 0x00007efeb0f57fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007efea4cbc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12856):
#0 0x00007efeb0f5bd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:37.357479 8863 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 9e690d05ade74ce38ef7c57ca1ff4709 and pid 12722
************************ BEGIN STACKS **************************
[New LWP 12723]
[New LWP 12724]
[New LWP 12725]
[New LWP 12726]
[New LWP 12732]
[New LWP 12733]
[New LWP 12734]
[New LWP 12737]
[New LWP 12738]
[New LWP 12739]
[New LWP 12740]
[New LWP 12741]
[New LWP 12742]
[New LWP 12743]
[New LWP 12744]
[New LWP 12745]
[New LWP 12746]
[New LWP 12747]
[New LWP 12748]
[New LWP 12749]
[New LWP 12750]
[New LWP 12751]
[New LWP 12752]
[New LWP 12753]
[New LWP 12754]
[New LWP 12755]
[New LWP 12756]
[New LWP 12757]
[New LWP 12758]
[New LWP 12759]
[New LWP 12760]
[New LWP 12761]
[New LWP 12762]
[New LWP 12763]
[New LWP 12764]
[New LWP 12765]
[New LWP 12766]
[New LWP 12767]
[New LWP 12768]
[New LWP 12769]
[New LWP 12770]
[New LWP 12771]
[New LWP 12772]
[New LWP 12773]
[New LWP 12774]
[New LWP 12775]
[New LWP 12776]
[New LWP 12777]
[New LWP 12778]
[New LWP 12779]
[New LWP 12780]
[New LWP 12781]
[New LWP 12782]
[New LWP 12783]
[New LWP 12784]
[New LWP 12785]
[New LWP 12786]
[New LWP 12787]
[New LWP 12788]
[New LWP 12789]
[New LWP 12790]
[New LWP 12791]
[New LWP 12792]
[New LWP 12793]
[New LWP 12794]
[New LWP 12795]
[New LWP 12796]
[New LWP 12797]
[New LWP 12798]
[New LWP 12799]
[New LWP 12800]
[New LWP 12801]
[New LWP 12802]
[New LWP 12803]
[New LWP 12804]
[New LWP 12805]
[New LWP 12806]
[New LWP 12807]
[New LWP 12808]
[New LWP 12809]
[New LWP 12810]
[New LWP 12811]
[New LWP 12812]
[New LWP 12813]
[New LWP 12814]
[New LWP 12815]
[New LWP 12816]
[New LWP 12817]
[New LWP 12818]
[New LWP 12819]
[New LWP 12820]
[New LWP 12821]
[New LWP 12822]
[New LWP 12823]
[New LWP 12824]
[New LWP 12825]
[New LWP 12826]
[New LWP 12827]
[New LWP 12828]
[New LWP 12829]
[New LWP 12830]
[New LWP 12831]
[New LWP 12832]
[New LWP 12833]
[New LWP 12834]
[New LWP 12835]
[New LWP 12836]
[New LWP 12837]
[New LWP 12838]
[New LWP 12839]
[New LWP 12840]
[New LWP 12841]
[New LWP 12842]
[New LWP 12843]
[New LWP 12844]
[New LWP 12845]
[New LWP 12846]
[New LWP 12847]
[New LWP 12848]
[New LWP 12849]
[New LWP 12850]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fc79d964d50 in ?? ()
Id Target Id Frame
* 1 LWP 12722 "kudu" 0x00007fc79d964d50 in ?? ()
2 LWP 12723 "kudu" 0x00007fc79d960fb9 in ?? ()
3 LWP 12724 "kudu" 0x00007fc79d960fb9 in ?? ()
4 LWP 12725 "kudu" 0x00007fc79d960fb9 in ?? ()
5 LWP 12726 "kernel-watcher-" 0x00007fc79d960fb9 in ?? ()
6 LWP 12732 "ntp client-1273" 0x00007fc79d9649e2 in ?? ()
7 LWP 12733 "file cache-evic" 0x00007fc79d960fb9 in ?? ()
8 LWP 12734 "sq_acceptor" 0x00007fc796686cb9 in ?? ()
9 LWP 12737 "rpc reactor-127" 0x00007fc796693a47 in ?? ()
10 LWP 12738 "rpc reactor-127" 0x00007fc796693a47 in ?? ()
11 LWP 12739 "rpc reactor-127" 0x00007fc796693a47 in ?? ()
12 LWP 12740 "rpc reactor-127" 0x00007fc796693a47 in ?? ()
13 LWP 12741 "MaintenanceMgr " 0x00007fc79d960ad3 in ?? ()
14 LWP 12742 "txn-status-mana" 0x00007fc79d960fb9 in ?? ()
15 LWP 12743 "collect_and_rem" 0x00007fc79d960fb9 in ?? ()
16 LWP 12744 "tc-session-exp-" 0x00007fc79d960fb9 in ?? ()
17 LWP 12745 "rpc worker-1274" 0x00007fc79d960ad3 in ?? ()
18 LWP 12746 "rpc worker-1274" 0x00007fc79d960ad3 in ?? ()
19 LWP 12747 "rpc worker-1274" 0x00007fc79d960ad3 in ?? ()
20 LWP 12748 "rpc worker-1274" 0x00007fc79d960ad3 in ?? ()
21 LWP 12749 "rpc worker-1274" 0x00007fc79d960ad3 in ?? ()
22 LWP 12750 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
23 LWP 12751 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
24 LWP 12752 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
25 LWP 12753 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
26 LWP 12754 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
27 LWP 12755 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
28 LWP 12756 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
29 LWP 12757 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
30 LWP 12758 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
31 LWP 12759 "rpc worker-1275" 0x00007fc79d960ad3 in ?? ()
32 LWP 12760 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
33 LWP 12761 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
34 LWP 12762 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
35 LWP 12763 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
36 LWP 12764 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
37 LWP 12765 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
38 LWP 12766 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
39 LWP 12767 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
40 LWP 12768 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
41 LWP 12769 "rpc worker-1276" 0x00007fc79d960ad3 in ?? ()
42 LWP 12770 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
43 LWP 12771 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
44 LWP 12772 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
45 LWP 12773 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
46 LWP 12774 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
47 LWP 12775 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
48 LWP 12776 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
49 LWP 12777 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
50 LWP 12778 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
51 LWP 12779 "rpc worker-1277" 0x00007fc79d960ad3 in ?? ()
52 LWP 12780 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
53 LWP 12781 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
54 LWP 12782 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
55 LWP 12783 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
56 LWP 12784 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
57 LWP 12785 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
58 LWP 12786 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
59 LWP 12787 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
60 LWP 12788 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
61 LWP 12789 "rpc worker-1278" 0x00007fc79d960ad3 in ?? ()
62 LWP 12790 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
63 LWP 12791 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
64 LWP 12792 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
65 LWP 12793 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
66 LWP 12794 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
67 LWP 12795 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
68 LWP 12796 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
69 LWP 12797 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
70 LWP 12798 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
71 LWP 12799 "rpc worker-1279" 0x00007fc79d960ad3 in ?? ()
72 LWP 12800 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
73 LWP 12801 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
74 LWP 12802 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
75 LWP 12803 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
76 LWP 12804 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
77 LWP 12805 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
78 LWP 12806 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
79 LWP 12807 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
80 LWP 12808 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
81 LWP 12809 "rpc worker-1280" 0x00007fc79d960ad3 in ?? ()
82 LWP 12810 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
83 LWP 12811 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
84 LWP 12812 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
85 LWP 12813 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
86 LWP 12814 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
87 LWP 12815 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
88 LWP 12816 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
89 LWP 12817 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
90 LWP 12818 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
91 LWP 12819 "rpc worker-1281" 0x00007fc79d960ad3 in ?? ()
92 LWP 12820 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
93 LWP 12821 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
94 LWP 12822 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
95 LWP 12823 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
96 LWP 12824 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
97 LWP 12825 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
98 LWP 12826 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
99 LWP 12827 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
100 LWP 12828 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
101 LWP 12829 "rpc worker-1282" 0x00007fc79d960ad3 in ?? ()
102 LWP 12830 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
103 LWP 12831 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
104 LWP 12832 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
105 LWP 12833 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
106 LWP 12834 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
107 LWP 12835 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
108 LWP 12836 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
109 LWP 12837 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
110 LWP 12838 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
111 LWP 12839 "rpc worker-1283" 0x00007fc79d960ad3 in ?? ()
112 LWP 12840 "rpc worker-1284" 0x00007fc79d960ad3 in ?? ()
113 LWP 12841 "rpc worker-1284" 0x00007fc79d960ad3 in ?? ()
114 LWP 12842 "rpc worker-1284" 0x00007fc79d960ad3 in ?? ()
115 LWP 12843 "rpc worker-1284" 0x00007fc79d960ad3 in ?? ()
116 LWP 12844 "rpc worker-1284" 0x00007fc79d960ad3 in ?? ()
117 LWP 12845 "diag-logger-128" 0x00007fc79d960fb9 in ?? ()
118 LWP 12846 "result-tracker-" 0x00007fc79d960fb9 in ?? ()
119 LWP 12847 "excess-log-dele" 0x00007fc79d960fb9 in ?? ()
120 LWP 12848 "acceptor-12848" 0x00007fc7966950c7 in ?? ()
121 LWP 12849 "heartbeat-12849" 0x00007fc79d960fb9 in ?? ()
122 LWP 12850 "maintenance_sch" 0x00007fc79d960fb9 in ?? ()
Thread 122 (LWP 12850):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc7529f06e0 in ?? ()
#2 0x000000000000005d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fc7529f0730 in ?? ()
#6 0x00000000000000ba in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12849):
#0 0x00007fc79d960fb9 in ?? ()
#1 0xb1139450f4d65aa7 in ?? ()
#2 0x0000000000000017 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007fc7532085f0 in ?? ()
#6 0x000000000000002e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 12848):
#0 0x00007fc7966950c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007fc753a2fd90 in ?? ()
#3 0x00007fc753a2fdc0 in ?? ()
#4 0x00007fc753a2fec0 in ?? ()
#5 0x00007fc753a2fdb0 in ?? ()
#6 0x00007fc753a2fe20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007fc753a2ff40 in ?? ()
#11 0x00007fc753a2f67c in ?? ()
#12 0x0000003253a2f5f0 in ?? ()
#13 0x00007fc753233000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 12847):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc754249f40 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 12846):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc754a62100 in ?? ()
#2 0x0000000000000017 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fc754a620f0 in ?? ()
#6 0x000000000000002e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12845):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12844):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fc755a91e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc755a91eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fc79d960770 in ?? ()
#10 0x00007fc755a91eb0 in ?? ()
#11 0x00000ff8eab523d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 12843):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000009 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fc7562a9e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc7562a9eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fc79d960770 in ?? ()
#10 0x00007fc7562a9eb0 in ?? ()
#11 0x00000ff8eac553d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 12842):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12841):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12840):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12839):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12838):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12837):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12836):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12835):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12834):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12833):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12832):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12831):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12830):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12829):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12828):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12827):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12826):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12825):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12824):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12823):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12822):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12821):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12820):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12819):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12818):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12817):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12816):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12815):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12814):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12813):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12812):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12811):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12810):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12809):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12808):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12807):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12806):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12805):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12804):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007fc769e62e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc769e62eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007fc79d960770 in ?? ()
#10 0x00007fc769e62eb0 in ?? ()
#11 0x00000ff8ed3cc5d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 12803):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 12802):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 12801):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12800):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12799):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12798):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12797):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12796):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12795):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12794):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12793):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 12792):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 12791):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 12790):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 12789):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 12788):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 12787):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 12786):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 12785):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 12784):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007fc774043e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc774043eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007fc79d960770 in ?? ()
#10 0x00007fc774043eb0 in ?? ()
#11 0x00000ff8ee8087d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 12783):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 12782):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 12781):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 12780):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 12779):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 12778):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 12777):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 12776):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 12775):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 12774):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 12773):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 12772):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 12771):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 12770):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 12769):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 12768):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 12767):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 12766):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 12765):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 12764):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fc77e224e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc77e224eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fc79d960770 in ?? ()
#10 0x00007fc77e224eb0 in ?? ()
#11 0x00000ff8efc449d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 12763):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 12762):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 12761):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 12760):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 12759):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 12758):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 12757):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 12756):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 12755):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 12754):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 12753):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 12752):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 12751):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 12750):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 12749):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 12748):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 12747):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 12746):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 12745):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 12744):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc788425cc0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007fc788425cb0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 12743):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 12742):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc789463240 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fc789463230 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 12741):
#0 0x00007fc79d960ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 12740):
#0 0x00007fc796693a47 in ?? ()
#1 0x00007fc78a4b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fc78a4b2310 in ?? ()
#4 0x00007fc78a4b2520 in ?? ()
#5 0x00007fc78a4b2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fc78a4b23e0 in ?? ()
#8 0x00007fc798d0625d in ?? ()
#9 0x3fb9859fc53b4000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000f6000000 in ?? ()
#14 0x00007fc700000000 in ?? ()
#15 0x41da17a845ddd36b in ?? ()
#16 0x00000ff97148e47c in ?? ()
#17 0x00007fc78a4b23c0 in ?? ()
#18 0x00007fc798d0aba3 in ?? ()
#19 0x00007fc78a4b2390 in ?? ()
#20 0x3fb9859fc53b4000 in ?? ()
#21 0x000000008a4b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9859fc53b4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 12739):
#0 0x00007fc796693a47 in ?? ()
#1 0x00007fc78acb3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fc78acb3310 in ?? ()
#4 0x00007fc78acb3520 in ?? ()
#5 0x00007fc78acb3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fc78acb33e0 in ?? ()
#8 0x00007fc798d0625d in ?? ()
#9 0x3fb959370ad66000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000c9000000 in ?? ()
#14 0x00007fc700000000 in ?? ()
#15 0x41da17a845ddd368 in ?? ()
#16 0x00000ff97158e67c in ?? ()
#17 0x00007fc78acb33c0 in ?? ()
#18 0x00007fc798d0aba3 in ?? ()
#19 0x00007fc78acb3390 in ?? ()
#20 0x3fb959370ad66000 in ?? ()
#21 0x000000008acb33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb959370ad66000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 12738):
#0 0x00007fc796693a47 in ?? ()
#1 0x00007fc78b4b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fc78b4b4310 in ?? ()
#4 0x00007fc78b4b4520 in ?? ()
#5 0x00007fc78b4b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fc78b4b43e0 in ?? ()
#8 0x00007fc798d0625d in ?? ()
#9 0x3fb970c318964000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000ce000000 in ?? ()
#14 0x00007fc700000000 in ?? ()
#15 0x41da17a845ddd36c in ?? ()
#16 0x00000ff97168e87c in ?? ()
#17 0x00007fc78b4b43c0 in ?? ()
#18 0x00007fc798d0aba3 in ?? ()
#19 0x00007fc78b4b4390 in ?? ()
#20 0x3fb970c318964000 in ?? ()
#21 0x000000008b4b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb970c318964000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 12737):
#0 0x00007fc796693a47 in ?? ()
#1 0x00007fc78d4ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fc78d4ad310 in ?? ()
#4 0x00007fc78d4ad520 in ?? ()
#5 0x00007fc78d4ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fc78d4ad3e0 in ?? ()
#8 0x00007fc798d0625d in ?? ()
#9 0x3fb97bfccc5b0000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000002f000000 in ?? ()
#14 0x00007fc700000000 in ?? ()
#15 0x41da17a845ddd369 in ?? ()
#16 0x00000ff971a8da7c in ?? ()
#17 0x00007fc78d4ad3c0 in ?? ()
#18 0x00007fc798d0aba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 12734):
#0 0x00007fc796686cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fc78eeb77c8 in ?? ()
#3 0x000060200001c9d0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 12733):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc78e6b60c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 12732):
#0 0x00007fc79d9649e2 in ?? ()
#1 0x00007fc78deb3ba0 in ?? ()
#2 0x00007fc78deb3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fc78deb3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fc78deb3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fc78deb3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fc7a3277edc in ?? ()
#12 0x00007fc7a32678d9 in ?? ()
#13 0x00007fc7a326bec6 in ?? ()
#14 0x00007fc7a3270900 in ?? ()
#15 0x00007fc7a3270729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fc79a53c4de in ?? ()
#18 0x00007fc79d95a6db in ?? ()
#19 0x00000ff971bce684 in ?? ()
#20 0x00007fc78deb3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12726):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x00007fc78feb8c80 in ?? ()
#2 0x0000000000000075 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007fc78feb8c70 in ?? ()
#6 0x00000000000000ea in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12725):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12724):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12723):
#0 0x00007fc79d960fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fc7916bc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12722):
#0 0x00007fc79d964d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:38.127071 8863 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 26e364b586e84dba94e1678b85d63274 and pid 13130
************************ BEGIN STACKS **************************
[New LWP 13131]
[New LWP 13132]
[New LWP 13133]
[New LWP 13134]
[New LWP 13140]
[New LWP 13141]
[New LWP 13142]
[New LWP 13145]
[New LWP 13146]
[New LWP 13147]
[New LWP 13148]
[New LWP 13149]
[New LWP 13150]
[New LWP 13152]
[New LWP 13153]
[New LWP 13154]
[New LWP 13155]
[New LWP 13156]
[New LWP 13157]
[New LWP 13158]
[New LWP 13159]
[New LWP 13160]
[New LWP 13161]
[New LWP 13162]
[New LWP 13163]
[New LWP 13164]
[New LWP 13165]
[New LWP 13166]
[New LWP 13167]
[New LWP 13168]
[New LWP 13169]
[New LWP 13170]
[New LWP 13171]
[New LWP 13172]
[New LWP 13173]
[New LWP 13174]
[New LWP 13175]
[New LWP 13176]
[New LWP 13177]
[New LWP 13178]
[New LWP 13179]
[New LWP 13180]
[New LWP 13181]
[New LWP 13182]
[New LWP 13183]
[New LWP 13184]
[New LWP 13185]
[New LWP 13186]
[New LWP 13187]
[New LWP 13188]
[New LWP 13189]
[New LWP 13190]
[New LWP 13191]
[New LWP 13192]
[New LWP 13193]
[New LWP 13194]
[New LWP 13195]
[New LWP 13196]
[New LWP 13197]
[New LWP 13198]
[New LWP 13199]
[New LWP 13200]
[New LWP 13201]
[New LWP 13202]
[New LWP 13203]
[New LWP 13204]
[New LWP 13205]
[New LWP 13206]
[New LWP 13207]
[New LWP 13209]
[New LWP 13210]
[New LWP 13211]
[New LWP 13212]
[New LWP 13213]
[New LWP 13214]
[New LWP 13215]
[New LWP 13216]
[New LWP 13217]
[New LWP 13218]
[New LWP 13219]
[New LWP 13220]
[New LWP 13221]
[New LWP 13222]
[New LWP 13223]
[New LWP 13224]
[New LWP 13225]
[New LWP 13226]
[New LWP 13227]
[New LWP 13228]
[New LWP 13229]
[New LWP 13230]
[New LWP 13231]
[New LWP 13232]
[New LWP 13233]
[New LWP 13234]
[New LWP 13235]
[New LWP 13236]
[New LWP 13237]
[New LWP 13238]
[New LWP 13239]
[New LWP 13240]
[New LWP 13241]
[New LWP 13242]
[New LWP 13243]
[New LWP 13244]
[New LWP 13245]
[New LWP 13246]
[New LWP 13247]
[New LWP 13248]
[New LWP 13249]
[New LWP 13250]
[New LWP 13251]
[New LWP 13252]
[New LWP 13253]
[New LWP 13254]
[New LWP 13255]
[New LWP 13256]
[New LWP 13257]
[New LWP 13258]
[New LWP 13259]
[New LWP 13260]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f18f7f70d50 in ?? ()
Id Target Id Frame
* 1 LWP 13130 "kudu" 0x00007f18f7f70d50 in ?? ()
2 LWP 13131 "kudu" 0x00007f18f7f6cfb9 in ?? ()
3 LWP 13132 "kudu" 0x00007f18f7f6cfb9 in ?? ()
4 LWP 13133 "kudu" 0x00007f18f7f6cfb9 in ?? ()
5 LWP 13134 "kernel-watcher-" 0x00007f18f7f6cfb9 in ?? ()
6 LWP 13140 "ntp client-1314" 0x00007f18f7f709e2 in ?? ()
7 LWP 13141 "file cache-evic" 0x00007f18f7f6cfb9 in ?? ()
8 LWP 13142 "sq_acceptor" 0x00007f18f0c92cb9 in ?? ()
9 LWP 13145 "rpc reactor-131" 0x00007f18f0c9fa47 in ?? ()
10 LWP 13146 "rpc reactor-131" 0x00007f18f0c9fa47 in ?? ()
11 LWP 13147 "rpc reactor-131" 0x00007f18f0c9fa47 in ?? ()
12 LWP 13148 "rpc reactor-131" 0x00007f18f0c9fa47 in ?? ()
13 LWP 13149 "MaintenanceMgr " 0x00007f18f7f6cad3 in ?? ()
14 LWP 13150 "txn-status-mana" 0x00007f18f7f6cfb9 in ?? ()
15 LWP 13152 "collect_and_rem" 0x00007f18f7f6cfb9 in ?? ()
16 LWP 13153 "tc-session-exp-" 0x00007f18f7f6cfb9 in ?? ()
17 LWP 13154 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
18 LWP 13155 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
19 LWP 13156 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
20 LWP 13157 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
21 LWP 13158 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
22 LWP 13159 "rpc worker-1315" 0x00007f18f7f6cad3 in ?? ()
23 LWP 13160 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
24 LWP 13161 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
25 LWP 13162 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
26 LWP 13163 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
27 LWP 13164 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
28 LWP 13165 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
29 LWP 13166 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
30 LWP 13167 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
31 LWP 13168 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
32 LWP 13169 "rpc worker-1316" 0x00007f18f7f6cad3 in ?? ()
33 LWP 13170 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
34 LWP 13171 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
35 LWP 13172 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
36 LWP 13173 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
37 LWP 13174 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
38 LWP 13175 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
39 LWP 13176 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
40 LWP 13177 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
41 LWP 13178 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
42 LWP 13179 "rpc worker-1317" 0x00007f18f7f6cad3 in ?? ()
43 LWP 13180 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
44 LWP 13181 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
45 LWP 13182 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
46 LWP 13183 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
47 LWP 13184 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
48 LWP 13185 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
49 LWP 13186 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
50 LWP 13187 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
51 LWP 13188 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
52 LWP 13189 "rpc worker-1318" 0x00007f18f7f6cad3 in ?? ()
53 LWP 13190 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
54 LWP 13191 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
55 LWP 13192 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
56 LWP 13193 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
57 LWP 13194 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
58 LWP 13195 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
59 LWP 13196 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
60 LWP 13197 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
61 LWP 13198 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
62 LWP 13199 "rpc worker-1319" 0x00007f18f7f6cad3 in ?? ()
63 LWP 13200 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
64 LWP 13201 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
65 LWP 13202 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
66 LWP 13203 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
67 LWP 13204 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
68 LWP 13205 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
69 LWP 13206 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
70 LWP 13207 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
71 LWP 13209 "rpc worker-1320" 0x00007f18f7f6cad3 in ?? ()
72 LWP 13210 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
73 LWP 13211 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
74 LWP 13212 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
75 LWP 13213 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
76 LWP 13214 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
77 LWP 13215 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
78 LWP 13216 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
79 LWP 13217 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
80 LWP 13218 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
81 LWP 13219 "rpc worker-1321" 0x00007f18f7f6cad3 in ?? ()
82 LWP 13220 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
83 LWP 13221 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
84 LWP 13222 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
85 LWP 13223 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
86 LWP 13224 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
87 LWP 13225 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
88 LWP 13226 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
89 LWP 13227 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
90 LWP 13228 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
91 LWP 13229 "rpc worker-1322" 0x00007f18f7f6cad3 in ?? ()
92 LWP 13230 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
93 LWP 13231 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
94 LWP 13232 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
95 LWP 13233 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
96 LWP 13234 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
97 LWP 13235 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
98 LWP 13236 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
99 LWP 13237 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
100 LWP 13238 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
101 LWP 13239 "rpc worker-1323" 0x00007f18f7f6cad3 in ?? ()
102 LWP 13240 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
103 LWP 13241 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
104 LWP 13242 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
105 LWP 13243 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
106 LWP 13244 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
107 LWP 13245 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
108 LWP 13246 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
109 LWP 13247 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
110 LWP 13248 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
111 LWP 13249 "rpc worker-1324" 0x00007f18f7f6cad3 in ?? ()
112 LWP 13250 "rpc worker-1325" 0x00007f18f7f6cad3 in ?? ()
113 LWP 13251 "rpc worker-1325" 0x00007f18f7f6cad3 in ?? ()
114 LWP 13252 "rpc worker-1325" 0x00007f18f7f6cad3 in ?? ()
115 LWP 13253 "rpc worker-1325" 0x00007f18f7f6cad3 in ?? ()
116 LWP 13254 "rpc worker-1325" 0x00007f18f7f6cad3 in ?? ()
117 LWP 13255 "diag-logger-132" 0x00007f18f7f6cfb9 in ?? ()
118 LWP 13256 "result-tracker-" 0x00007f18f7f6cfb9 in ?? ()
119 LWP 13257 "excess-log-dele" 0x00007f18f7f6cfb9 in ?? ()
120 LWP 13258 "acceptor-13258" 0x00007f18f0ca10c7 in ?? ()
121 LWP 13259 "heartbeat-13259" 0x00007f18f7f6cfb9 in ?? ()
122 LWP 13260 "maintenance_sch" 0x00007f18f7f6cfb9 in ?? ()
Thread 122 (LWP 13260):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18acfbc6e0 in ?? ()
#2 0x0000000000000051 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f18acfbc730 in ?? ()
#6 0x00000000000000a2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 13259):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0xb1139450f4d65aa7 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f18ad7d45f0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 13258):
#0 0x00007f18f0ca10c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f18adffbd90 in ?? ()
#3 0x00007f18adffbdc0 in ?? ()
#4 0x00007f18adffbec0 in ?? ()
#5 0x00007f18adffbdb0 in ?? ()
#6 0x00007f18adffbe20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f18adffbf40 in ?? ()
#11 0x00007f18adffb67c in ?? ()
#12 0x00000032adffb5f0 in ?? ()
#13 0x00007f18ad7ff000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 13257):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18ae814f40 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 13256):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18af02e100 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f18af02e0f0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 13255):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 13254):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b6848 in ?? ()
#4 0x00007f18b005de90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18b005deb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 13253):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f18b0875e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18b0875eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 13252):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 13251):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 13250):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 13249):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 13248):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 13247):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 13246):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 13245):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 13244):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 13243):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 13242):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 13241):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 13240):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 13239):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 13238):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 13237):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 13236):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 13235):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 13234):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 13233):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 13232):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 13231):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 13230):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 13229):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 13228):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 13227):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 13226):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 13225):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 13224):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 13223):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 13222):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 13221):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 13220):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 13219):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 13218):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 13217):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 13216):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 13215):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 13214):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x000000000000012c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f18c442ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18c442eeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 13213):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000102 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f18c4c46e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18c4c46eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 13212):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 13211):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 13210):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 13209):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 13207):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 13206):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 13205):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 13204):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 13203):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 13202):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 13201):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 13200):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 13199):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 13198):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 13197):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 13196):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 13195):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 13194):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 13193):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f18cddf7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18cddf7eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f18f7f6c770 in ?? ()
#10 0x00007f18cddf7eb0 in ?? ()
#11 0x00000fe319bbefd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 13192):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 13191):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 13190):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 13189):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 13188):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 13187):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 13186):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 13185):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 13184):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 13183):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 13182):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 13181):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 13180):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 13179):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 13178):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 13177):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 13176):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 13175):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 13174):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 13173):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000023 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f18d7fd8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18d7fd8eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f18f7f6c770 in ?? ()
#10 0x00007f18d7fd8eb0 in ?? ()
#11 0x00000fe31affb1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 13172):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x000000000000003d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f18d87f0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18d87f0eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f18f7f6c770 in ?? ()
#10 0x00007f18d87f0eb0 in ?? ()
#11 0x00000fe31b0fe1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 13171):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 13170):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 13169):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 13168):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 13167):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 13166):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 13165):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 13164):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 13163):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 13162):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 13161):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 13160):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 13159):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 13158):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 13157):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 13156):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 13155):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 13154):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 13153):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18e21dccc0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007f18e21dccb0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 13152):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 13150):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18e3a71240 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f18e3a71230 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 13149):
#0 0x00007f18f7f6cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 13148):
#0 0x00007f18f0c9fa47 in ?? ()
#1 0x00007f18e4ab2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f18e4ab2310 in ?? ()
#4 0x00007f18e4ab2520 in ?? ()
#5 0x00007f18e4ab2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f18e4ab23e0 in ?? ()
#8 0x00007f18f331225d in ?? ()
#9 0x3fb9806105d12000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000be000000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41da17a845ddd36a in ?? ()
#16 0x00000fe39c94e47c in ?? ()
#17 0x00007f18e4ab23c0 in ?? ()
#18 0x00007f18f3316ba3 in ?? ()
#19 0x00007f18e4ab2390 in ?? ()
#20 0x3fb9806105d12000 in ?? ()
#21 0x00000000e4ab23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9806105d12000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 13147):
#0 0x00007f18f0c9fa47 in ?? ()
#1 0x00007f18e52b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f18e52b3310 in ?? ()
#4 0x00007f18e52b3520 in ?? ()
#5 0x00007f18e52b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f18e52b33e0 in ?? ()
#8 0x00007f18f331225d in ?? ()
#9 0x3fb989c57ba50000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000ca800000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41da17a845ddd36b in ?? ()
#16 0x00000fe39ca4e67c in ?? ()
#17 0x00007f18e52b33c0 in ?? ()
#18 0x00007f18f3316ba3 in ?? ()
#19 0x00007f18e52b3390 in ?? ()
#20 0x3fb989c57ba50000 in ?? ()
#21 0x00000000e52b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb989c57ba50000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 13146):
#0 0x00007f18f0c9fa47 in ?? ()
#1 0x00007f18e5ab4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f18e5ab4310 in ?? ()
#4 0x00007f18e5ab4520 in ?? ()
#5 0x00007f18e5ab4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f18e5ab43e0 in ?? ()
#8 0x00007f18f331225d in ?? ()
#9 0x3fb95c27552b4000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000a3000000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41da17a845ddd367 in ?? ()
#16 0x00000fe39cb4e87c in ?? ()
#17 0x00007f18e5ab43c0 in ?? ()
#18 0x00007f18f3316ba3 in ?? ()
#19 0x00007f18e5ab4390 in ?? ()
#20 0x3fb95c27552b4000 in ?? ()
#21 0x00000000e5ab43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb95c27552b4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 13145):
#0 0x00007f18f0c9fa47 in ?? ()
#1 0x00007f18e7aad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f18e7aad310 in ?? ()
#4 0x00007f18e7aad520 in ?? ()
#5 0x00007f18e7aad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f18e7aad3e0 in ?? ()
#8 0x00007f18f331225d in ?? ()
#9 0x3f9de427c38f8000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000e2200000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41da17a845ddd368 in ?? ()
#16 0x00000fe39cf4da7c in ?? ()
#17 0x00007f18e7aad3c0 in ?? ()
#18 0x00007f18f3316ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 13142):
#0 0x00007f18f0c92cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f18e94b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 13141):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18e8cb60c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 13140):
#0 0x00007f18f7f709e2 in ?? ()
#1 0x00007f18e84b3ba0 in ?? ()
#2 0x00007f18e84b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f18e84b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f18e84b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f18e84b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f18fd883edc in ?? ()
#12 0x00007f18fd8738d9 in ?? ()
#13 0x00007f18fd877ec6 in ?? ()
#14 0x00007f18fd87c900 in ?? ()
#15 0x00007f18fd87c729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f18f4b484de in ?? ()
#18 0x00007f18f7f666db in ?? ()
#19 0x00000fe39d08e684 in ?? ()
#20 0x00007f18e84b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 13134):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x00007f18ea4b8c80 in ?? ()
#2 0x000000000000006a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f18ea4b8c70 in ?? ()
#6 0x00000000000000d4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 13133):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 13132):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 13131):
#0 0x00007f18f7f6cfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f18ebcbc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 13130):
#0 0x00007f18f7f70d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:38.891875 8863 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 2cbadb2aefb24b38b5dca68b10368ddf and pid 12992
************************ BEGIN STACKS **************************
[New LWP 12994]
[New LWP 12995]
[New LWP 12996]
[New LWP 12997]
[New LWP 13003]
[New LWP 13004]
[New LWP 13005]
[New LWP 13008]
[New LWP 13009]
[New LWP 13010]
[New LWP 13011]
[New LWP 13012]
[New LWP 13013]
[New LWP 13015]
[New LWP 13016]
[New LWP 13017]
[New LWP 13018]
[New LWP 13019]
[New LWP 13020]
[New LWP 13021]
[New LWP 13022]
[New LWP 13023]
[New LWP 13024]
[New LWP 13025]
[New LWP 13026]
[New LWP 13027]
[New LWP 13028]
[New LWP 13029]
[New LWP 13030]
[New LWP 13031]
[New LWP 13032]
[New LWP 13033]
[New LWP 13034]
[New LWP 13035]
[New LWP 13036]
[New LWP 13037]
[New LWP 13038]
[New LWP 13039]
[New LWP 13040]
[New LWP 13041]
[New LWP 13042]
[New LWP 13043]
[New LWP 13044]
[New LWP 13045]
[New LWP 13046]
[New LWP 13047]
[New LWP 13048]
[New LWP 13049]
[New LWP 13050]
[New LWP 13051]
[New LWP 13052]
[New LWP 13053]
[New LWP 13054]
[New LWP 13055]
[New LWP 13056]
[New LWP 13057]
[New LWP 13058]
[New LWP 13059]
[New LWP 13060]
[New LWP 13061]
[New LWP 13062]
[New LWP 13063]
[New LWP 13064]
[New LWP 13065]
[New LWP 13066]
[New LWP 13067]
[New LWP 13068]
[New LWP 13069]
[New LWP 13070]
[New LWP 13071]
[New LWP 13072]
[New LWP 13073]
[New LWP 13074]
[New LWP 13075]
[New LWP 13076]
[New LWP 13077]
[New LWP 13078]
[New LWP 13079]
[New LWP 13080]
[New LWP 13081]
[New LWP 13082]
[New LWP 13083]
[New LWP 13084]
[New LWP 13085]
[New LWP 13086]
[New LWP 13087]
[New LWP 13088]
[New LWP 13089]
[New LWP 13090]
[New LWP 13091]
[New LWP 13092]
[New LWP 13093]
[New LWP 13094]
[New LWP 13095]
[New LWP 13096]
[New LWP 13097]
[New LWP 13098]
[New LWP 13099]
[New LWP 13100]
[New LWP 13101]
[New LWP 13102]
[New LWP 13103]
[New LWP 13104]
[New LWP 13105]
[New LWP 13106]
[New LWP 13107]
[New LWP 13108]
[New LWP 13109]
[New LWP 13110]
[New LWP 13111]
[New LWP 13112]
[New LWP 13113]
[New LWP 13114]
[New LWP 13115]
[New LWP 13116]
[New LWP 13117]
[New LWP 13118]
[New LWP 13119]
[New LWP 13120]
[New LWP 13121]
[New LWP 13122]
[New LWP 13360]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f7a5d01fd50 in ?? ()
Id Target Id Frame
* 1 LWP 12992 "kudu" 0x00007f7a5d01fd50 in ?? ()
2 LWP 12994 "kudu" 0x00007f7a5d01bfb9 in ?? ()
3 LWP 12995 "kudu" 0x00007f7a5d01bfb9 in ?? ()
4 LWP 12996 "kudu" 0x00007f7a5d01bfb9 in ?? ()
5 LWP 12997 "kernel-watcher-" 0x00007f7a5d01bfb9 in ?? ()
6 LWP 13003 "ntp client-1300" 0x00007f7a5d01f9e2 in ?? ()
7 LWP 13004 "file cache-evic" 0x00007f7a5d01bfb9 in ?? ()
8 LWP 13005 "sq_acceptor" 0x00007f7a55d41cb9 in ?? ()
9 LWP 13008 "rpc reactor-130" 0x00007f7a55d4ea47 in ?? ()
10 LWP 13009 "rpc reactor-130" 0x00007f7a55d4ea47 in ?? ()
11 LWP 13010 "rpc reactor-130" 0x00007f7a55d4ea47 in ?? ()
12 LWP 13011 "rpc reactor-130" 0x00007f7a55d4ea47 in ?? ()
13 LWP 13012 "MaintenanceMgr " 0x00007f7a5d01bad3 in ?? ()
14 LWP 13013 "txn-status-mana" 0x00007f7a5d01bfb9 in ?? ()
15 LWP 13015 "collect_and_rem" 0x00007f7a5d01bfb9 in ?? ()
16 LWP 13016 "tc-session-exp-" 0x00007f7a5d01bfb9 in ?? ()
17 LWP 13017 "rpc worker-1301" 0x00007f7a5d01bad3 in ?? ()
18 LWP 13018 "rpc worker-1301" 0x00007f7a5d01bad3 in ?? ()
19 LWP 13019 "rpc worker-1301" 0x00007f7a5d01bad3 in ?? ()
20 LWP 13020 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
21 LWP 13021 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
22 LWP 13022 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
23 LWP 13023 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
24 LWP 13024 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
25 LWP 13025 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
26 LWP 13026 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
27 LWP 13027 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
28 LWP 13028 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
29 LWP 13029 "rpc worker-1302" 0x00007f7a5d01bad3 in ?? ()
30 LWP 13030 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
31 LWP 13031 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
32 LWP 13032 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
33 LWP 13033 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
34 LWP 13034 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
35 LWP 13035 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
36 LWP 13036 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
37 LWP 13037 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
38 LWP 13038 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
39 LWP 13039 "rpc worker-1303" 0x00007f7a5d01bad3 in ?? ()
40 LWP 13040 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
41 LWP 13041 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
42 LWP 13042 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
43 LWP 13043 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
44 LWP 13044 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
45 LWP 13045 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
46 LWP 13046 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
47 LWP 13047 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
48 LWP 13048 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
49 LWP 13049 "rpc worker-1304" 0x00007f7a5d01bad3 in ?? ()
50 LWP 13050 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
51 LWP 13051 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
52 LWP 13052 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
53 LWP 13053 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
54 LWP 13054 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
55 LWP 13055 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
56 LWP 13056 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
57 LWP 13057 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
58 LWP 13058 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
59 LWP 13059 "rpc worker-1305" 0x00007f7a5d01bad3 in ?? ()
60 LWP 13060 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
61 LWP 13061 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
62 LWP 13062 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
63 LWP 13063 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
64 LWP 13064 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
65 LWP 13065 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
66 LWP 13066 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
67 LWP 13067 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
68 LWP 13068 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
69 LWP 13069 "rpc worker-1306" 0x00007f7a5d01bad3 in ?? ()
70 LWP 13070 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
71 LWP 13071 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
72 LWP 13072 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
73 LWP 13073 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
74 LWP 13074 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
75 LWP 13075 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
76 LWP 13076 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
77 LWP 13077 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
78 LWP 13078 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
79 LWP 13079 "rpc worker-1307" 0x00007f7a5d01bad3 in ?? ()
80 LWP 13080 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
81 LWP 13081 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
82 LWP 13082 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
83 LWP 13083 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
84 LWP 13084 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
85 LWP 13085 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
86 LWP 13086 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
87 LWP 13087 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
88 LWP 13088 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
89 LWP 13089 "rpc worker-1308" 0x00007f7a5d01bad3 in ?? ()
90 LWP 13090 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
91 LWP 13091 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
92 LWP 13092 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
93 LWP 13093 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
94 LWP 13094 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
95 LWP 13095 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
96 LWP 13096 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
97 LWP 13097 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
98 LWP 13098 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
99 LWP 13099 "rpc worker-1309" 0x00007f7a5d01bad3 in ?? ()
100 LWP 13100 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
101 LWP 13101 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
102 LWP 13102 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
103 LWP 13103 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
104 LWP 13104 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
105 LWP 13105 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
106 LWP 13106 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
107 LWP 13107 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
108 LWP 13108 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
109 LWP 13109 "rpc worker-1310" 0x00007f7a5d01bad3 in ?? ()
110 LWP 13110 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
111 LWP 13111 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
112 LWP 13112 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
113 LWP 13113 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
114 LWP 13114 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
115 LWP 13115 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
116 LWP 13116 "rpc worker-1311" 0x00007f7a5d01bad3 in ?? ()
117 LWP 13117 "diag-logger-131" 0x00007f7a5d01bfb9 in ?? ()
118 LWP 13118 "result-tracker-" 0x00007f7a5d01bfb9 in ?? ()
119 LWP 13119 "excess-log-dele" 0x00007f7a5d01bfb9 in ?? ()
120 LWP 13120 "acceptor-13120" 0x00007f7a55d500c7 in ?? ()
121 LWP 13121 "heartbeat-13121" 0x00007f7a5d01bfb9 in ?? ()
122 LWP 13122 "maintenance_sch" 0x00007f7a5d01bfb9 in ?? ()
123 LWP 13360 "raft [worker]-1" 0x00007f7a5d01bfb9 in ?? ()
Thread 123 (LWP 13360):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000003ce in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f7a0caa6ba0 in ?? ()
#5 0x00007f7a0caa6830 in ?? ()
#6 0x000000000000079c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 13122):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a118536e0 in ?? ()
#2 0x000000000000005b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f7a11853730 in ?? ()
#6 0x00000000000000b6 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 13121):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0xb1139450f4d65aa7 in ?? ()
#2 0x0000000000000019 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f7a1206b5f0 in ?? ()
#6 0x0000000000000032 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 13120):
#0 0x00007f7a55d500c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f7a12892d90 in ?? ()
#3 0x00007f7a12892dc0 in ?? ()
#4 0x00007f7a12892ec0 in ?? ()
#5 0x00007f7a12892db0 in ?? ()
#6 0x00007f7a12892e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f7a12892f40 in ?? ()
#11 0x00007f7a1289267c in ?? ()
#12 0x00000032128925f0 in ?? ()
#13 0x00007f7a12096000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 13119):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a130abf40 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 13118):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a138c5100 in ?? ()
#2 0x0000000000000017 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f7a138c50f0 in ?? ()
#6 0x000000000000002e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 13117):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 13116):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b6848 in ?? ()
#4 0x00007f7a148f4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a148f4eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 13115):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f7a1510ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a1510ceb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 13114):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 13113):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 13112):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 13111):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 13110):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 13109):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 13108):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 13107):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 13106):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 13105):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 13104):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 13103):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 13102):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 13101):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 13100):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 13099):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 13098):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 13097):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 13096):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 13095):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 13094):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 13093):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 13092):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 13091):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 13090):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 13089):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 13088):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 13087):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 13086):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 13085):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 13084):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 13083):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 13082):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 13081):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 13080):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 13079):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 13078):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 13077):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 13076):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f7a28cc5e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a28cc5eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 13075):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 13074):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 13073):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 13072):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 13071):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 13070):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 13069):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 13068):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 13067):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 13066):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 13065):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 13064):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 13063):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 13062):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 13061):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 13060):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 13059):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 13058):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 13057):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 13056):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f7a32ea6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a32ea6eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f7a5d01b770 in ?? ()
#10 0x00007f7a32ea6eb0 in ?? ()
#11 0x00000fef465d4dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 13055):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 13054):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 13053):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 13052):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 13051):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 13050):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 13049):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 13048):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 13047):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 13046):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 13045):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 13044):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 13043):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 13042):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 13041):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 13040):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 13039):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 13038):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 13037):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 13036):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x00000000000000a2 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007f7a3d087e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a3d087eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 13035):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x00000000000000c7 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f7a3d89fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a3d89feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f7a5d01b770 in ?? ()
#10 0x00007f7a3d89feb0 in ?? ()
#11 0x00000fef47b13fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 13034):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000134 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007f7a3e0b7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7a3e0b7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 13033):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 13032):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 13031):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 13030):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 13029):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 13028):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 13027):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 13026):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 13025):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 13024):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 13023):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 13022):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 13021):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 13020):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 13019):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 13018):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 13017):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 13016):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a47295cc0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007f7a47295cb0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 13015):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 13013):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a48b1a240 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f7a48b1a230 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 13012):
#0 0x00007f7a5d01bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 13011):
#0 0x00007f7a55d4ea47 in ?? ()
#1 0x00007f7a49b69320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f7a49b69310 in ?? ()
#4 0x00007f7a49b69520 in ?? ()
#5 0x00007f7a49b69360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f7a49b693e0 in ?? ()
#8 0x00007f7a583c125d in ?? ()
#9 0x3fb98a5ecb0ee000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000eb800000 in ?? ()
#14 0x00007f7a00000000 in ?? ()
#15 0x41da17a845ddd368 in ?? ()
#16 0x00000fefc936527c in ?? ()
#17 0x00007f7a49b693c0 in ?? ()
#18 0x00007f7a583c5ba3 in ?? ()
#19 0x00007f7a49b69390 in ?? ()
#20 0x3fb98a5ecb0ee000 in ?? ()
#21 0x0000000049b693e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98a5ecb0ee000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 13010):
#0 0x00007f7a55d4ea47 in ?? ()
#1 0x00007f7a4a380320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f7a4a380310 in ?? ()
#4 0x00007f7a4a380520 in ?? ()
#5 0x00007f7a4a380360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f7a4a3803e0 in ?? ()
#8 0x00007f7a583c125d in ?? ()
#9 0x3fa1c1d30d728000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000a0000000 in ?? ()
#14 0x00007f7a00000000 in ?? ()
#15 0x41da17a845ddd36a in ?? ()
#16 0x00000fefc946807c in ?? ()
#17 0x00007f7a4a3803c0 in ?? ()
#18 0x00007f7a583c5ba3 in ?? ()
#19 0x00007f7a4a380390 in ?? ()
#20 0x3fa1c1d30d728000 in ?? ()
#21 0x000000004a3803e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fa1c1d30d728000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 13009):
#0 0x00007f7a55d4ea47 in ?? ()
#1 0x00007f7a4ab97320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f7a4ab97310 in ?? ()
#4 0x00007f7a4ab97520 in ?? ()
#5 0x00007f7a4ab97360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f7a4ab973e0 in ?? ()
#8 0x00007f7a583c125d in ?? ()
#9 0x3fa65f68e0020000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000c8000000 in ?? ()
#14 0x00007f7a00000000 in ?? ()
#15 0x41da17a845ddd369 in ?? ()
#16 0x00000fefc956ae7c in ?? ()
#17 0x00007f7a4ab973c0 in ?? ()
#18 0x00007f7a583c5ba3 in ?? ()
#19 0x00007f7a4ab97390 in ?? ()
#20 0x3fa65f68e0020000 in ?? ()
#21 0x000000004ab973e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fa65f68e0020000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 13008):
#0 0x00007f7a55d4ea47 in ?? ()
#1 0x00007f7a4cbad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f7a4cbad310 in ?? ()
#4 0x00007f7a4cbad520 in ?? ()
#5 0x00007f7a4cbad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f7a4cbad3e0 in ?? ()
#8 0x00007f7a583c125d in ?? ()
#9 0x3fb9720bc2ff0000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000029000000 in ?? ()
#14 0x00007f7a00000000 in ?? ()
#15 0x41da17a845ddd36a in ?? ()
#16 0x00000fefc996da7c in ?? ()
#17 0x00007f7a4cbad3c0 in ?? ()
#18 0x00007f7a583c5ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 13005):
#0 0x00007f7a55d41cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f7a4e5b77c8 in ?? ()
#3 0x000060200001d790 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 13004):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a4ddb60c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 13003):
#0 0x00007f7a5d01f9e2 in ?? ()
#1 0x00007f7a4d5b3ba0 in ?? ()
#2 0x00007f7a4d5b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f7a4d5b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f7a4d5b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f7a4d5b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f7a62932edc in ?? ()
#12 0x00007f7a629228d9 in ?? ()
#13 0x00007f7a62926ec6 in ?? ()
#14 0x00007f7a6292b900 in ?? ()
#15 0x00007f7a6292b729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f7a59bf74de in ?? ()
#18 0x00007f7a5d0156db in ?? ()
#19 0x00000fefc9aae684 in ?? ()
#20 0x00007f7a4d5b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12997):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x00007f7a4f5b8c80 in ?? ()
#2 0x0000000000000074 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f7a4f5b8c70 in ?? ()
#6 0x00000000000000e8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12996):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12995):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12994):
#0 0x00007f7a5d01bfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f7a50dbc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12992):
#0 0x00007f7a5d01fd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:39.646373 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 12856
I20250627 13:57:39.689989 12979 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:40.024924 13510 raft_consensus.cc:491] T 648e63fd67a54d22afac864a578a6912 P 7397f03f0d404d6692a8318392f3ba7c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 2cbadb2aefb24b38b5dca68b10368ddf)
I20250627 13:57:40.025064 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 12856
W20250627 13:57:40.074184 13009 connection.cc:537] client connection to 127.8.167.193:37697 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:40.074646 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 12722
W20250627 13:57:40.074589 13009 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:40.076619 13009 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 7397f03f0d404d6692a8318392f3ba7c (127.8.167.193:37697): Couldn't send request to peer 7397f03f0d404d6692a8318392f3ba7c. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.193:37697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:40.081404 12843 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:40.174540 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 12722
I20250627 13:57:40.195703 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 13130
I20250627 13:57:40.202694 13254 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:40.536940 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 13130
W20250627 13:57:40.590286 13011 connection.cc:537] client connection to 127.8.167.195:34991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:40.590672 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 12992
W20250627 13:57:40.591643 13011 consensus_peers.cc:487] T 648e63fd67a54d22afac864a578a6912 P 2cbadb2aefb24b38b5dca68b10368ddf -> Peer 26e364b586e84dba94e1678b85d63274 (127.8.167.195:34991): Couldn't send request to peer 26e364b586e84dba94e1678b85d63274. Status: Network error: Client connection negotiation failed: client connection to 127.8.167.195:34991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:40.597002 13116 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:40.929711 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 12992
W20250627 13:57:40.984581 10079 connection.cc:537] server connection from 127.8.167.196:39735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:40.984968 8863 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu pid 10064
I20250627 13:57:40.991271 10124 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:41.142314 8863 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskHxmGc1/build/asan/bin/kudu with pid 10064
2025-06-27T13:57:41Z chronyd exiting
I20250627 13:57:41.188884 8863 test_util.cc:183] -----------------------------------------------
I20250627 13:57:41.188977 8863 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskHxmGc1/test-tmp/maintenance_mode-itest.0.RollingRestartArgs_RollingRestartITest.TestWorkloads_4.1751032515993529-8863-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 04-00 00-00 BE-BE BE-BE A0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-27 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 04-00 00-00 03-00 00-00 01-BE BE-BE BE-BE BE-BE> (98166 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (98166 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test suites ran. (145132 ms total)
[ PASSED ] 1 test.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/4, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 04-00 00-00 BE-BE BE-BE A0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-24 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-27 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 04-00 00-00 03-00 00-00 01-BE BE-BE BE-BE BE-BE>
1 FAILED TEST